Systems and methods for classification of arterial image regions and features thereof

ABSTRACT

In part, the disclosure relates to methods, and systems suitable for evaluating image data from a patient on a real time or substantially real time basis using machine learning (ML) methods and systems. Systems and methods for improving diagnostic tools for end users such as cardiologists and imaging specialists using machine learning techniques applied to specific problems associated with intravascular images that have polar representations. Further, given the use of rotating probes to obtain image data for OCT, IVUS, and other imaging data, dealing with the two coordinate systems associated therewith creates challenges. The present disclosure addresses these and numerous other challenges relating to solving the problem of quickly imaging and diagnosis a patient such that stenting and other procedures may be applied during a single session in the cath lab.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 62/791,876 filed on Jan. 13, 2019, the entiredisclosure of which is incorporated by reference therein.

FIELD

In part, this application relates to imaging arteries and segmenting andcharacterizing components thereof. Specifically, in some embodiments,this application relates to applications of machine learning tocharacterizing and/or classifying arterial tissue and related arterialregions and features of interest.

BACKGROUND

Optical coherence tomography (OCT) is an interferometric imagingtechnique with widespread applications in ophthalmology, cardiology,gastroenterology and other fields of medicine. The ability to viewsubsurface structures with high resolution through small-diameterfiber-optic probes makes OCT especially useful for minimally invasiveimaging of internal tissues and organs. OCT systems can generate imagesup to 100 frames per second, making it possible to image coronaryarteries in the beating heart artery within a few seconds. OCT can beimplemented in both time domain (TD-OCT) and frequency domain (Fourierdomain OCT or optical frequency domain imaging, OFDI). OCT can be usedin conjunction with various other imaging technologies such asintravascular ultrasound (IVUS), angiography, fluoroscopy, x-ray-basedimaging systems, and other imaging technologies.

OCT imaging of portions of a patient's body provides a useful tool fordoctors to determine the best type and course of treatment. For example,imaging of coronary arteries by intravascular OCT may reveal thelocation of a stenosis, the presence of vulnerable plaques, or the typeof atherosclerotic plaque. This information helps cardiologists choosewhich treatment would best serve the patient—drug therapy (e.g.,cholesterol-lowering medication), a catheter-based therapy likeangioplasty and stenting, or an invasive surgical procedure likecoronary bypass surgery. In addition to its applications in clinicalmedicine, OCT is also very useful for drug development in animal andclinical trials.

Normal arteries have a consistent layered structure consisting ofintima, media and adventitia. As a result of the process ofatherosclerosis, the intima becomes pathologically thickened and maycontain plaques composed of different types of tissues, including fiber,proteoglycans, lipid and calcium, as well as macrophages and otherinflammatory cells. These tissue types have different optical propertiesthat can be measured by manual measurements and imaging technologies.The plaques that are believed to be most pathologically significant arethe so-called vulnerable plaques that have a fibrous cap with anunderlying lipid pool.

In a typical OCT imaging system, an optical probe mounted on a catheteris carefully maneuvered to a point of interest such as within a coronaryblood vessel. The optical beams are then transmitted and thebackscattered signals are received through coherent detection using aninterferometer. As the probe is scanned through a predetermined line orarea, many data lines can be collected. An image (2D or 3D) is thenreconstructed using well-known techniques. This image is then analyzedvisually by a cardiologist to assess pathological features, such asvessel wall thickening and plaque composition.

Since tissue type is identified by its appearance on the screen, errorsmay occur in the analysis because certain information (such as tissuetype) cannot be readily discerned. Various other components or regionsof interest with regard to a given patient artery, organ, or other bodypart that are difficult to accurately classify by visual inspectionoccur in virtually all branches of medicine. A need therefore exists forsystems and methods of detecting various targets in image data andrepresenting the same to end users. The present disclosure addressesthese challenges and others.

SUMMARY

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of includes instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions. Onegeneral aspect includes a method of assessing a coronary artery usingone or more machine learning systems.

In one embodiment, the disclosure relates to various methods steps. Themethod may include acquiring a set of image data comprising frames ofpolar images; annotating one or more regions or features of interest ineach polar image of the set of images such that each annotated region orfeature is a ground truth annotation; training a neural network of amachine learning system using set of annotated polar images, whereineach a plurality of regions in each polar region are identified byclass; inputting polar image data to the trained neural network; anddisplaying predictive output images, wherein predictive output imagescomprise color coded regions, wherein each color corresponds to a class.

In one embodiment, the system includes one or more AI processors havingan associated memory, wherein one or more trained software-based neuralnetworks executes on one or more AI processors. The machine learningsystem can include a convolutional neural network. The method mayinclude acquiring a set of image data, such as arterial image data. Inone embodiment, the image data includes intravascular image data. In oneembodiment, the image data includes polar images. The method may includeannotating one or more regions or features of interest in each polarimage of the set of images such that each annotated region or feature isa ground truth annotation. The ground truth annotations may be stored inmemory as a set of ground truth masks. The method may include training aneural network of a machine learning system using set of annotated polarimages, such as through the set of ground truth masks. In oneembodiment, one ground truth mask includes a region of interestcorresponding to a particular feature or channel

Thus, a first region of a ground truth mask may correspond to a firstfeature and a second region of a ground truth mask may correspond to asecond feature. The foregoing may be applied to M features and Pregions, wherein each feature corresponds to one or more regions. As anexample, two or more regions of calcium, a region of lumen, and a regionof intima may be part of one ground truth mask, while the classificationthereof by class or type would be calcium, lumen and intima, with eachregion so classified/segmented. In one embodiment, ground truth masksare used to train a neural network to detect/predict which regions ininput image data from a user correspond to a particular feature orchannel. In one embodiment, the method includes inputting image data,such as polar image data and/or ground truth masks to the neural networkto train the neural network. The neural network is trained over one ormore epochs until it operably detects multiple features/channels withinan error threshold. In one embodiment, the method includes inputtingimage data, such as polar image data to the trained neural network anddisplaying predictive output images from a machine learning system. Inone embodiment, ground truth image masks are generated in response toannotating a depiction of an image in a user interface. In variousembodiments, a polar image is annotated in such a user interface. Theneural network is implemented in one or more software applications suchas PyTorch, LibTorch, and others disclosed herein. Other embodiments ofthis aspect include corresponding computer systems, apparatus, andcomputer programs recorded on one or more computer storage devices, eachconfigured to perform the actions of the methods.

Implementations may include one or more of the following processes orsteps. In one embodiment, each image includes a plurality of image dataelements with respect to the coronary artery. In one embodiment,annotating is performed with a graphical user interface that includesuser controls to select groups of pixels or a two-dimensional boundaryto define features of interest. In one embodiment, the training of theneural network is repeated until an output of a cost function is at orbelow a threshold, wherein the cost function compares predictive outputsof an MLS with ground truth inputs. In various embodiments, a crossentropy assessment is used to measure error associated with costfunction. The method may further include classifying the one or moreregions or features of interest for each polar image as a type or class.In one embodiment, the type or class is selected from the group includesintima, media, adventitia, lumen, EEL, IEL plaque, calcium, calciumplaques. In one embodiment, the image data used with systems and methodsdisclosed herein includes carpet view images, scan lines, pixels, 2Dimages, 3D images, angiography images, intravascular images, CT scanimages, x-ray images, and other images of arteries, veins, organs orother components of the circulatory system. The foregoing features,regions, channels, classes, etc. may be detected using a neural networktrained relative thereto.

In one embodiment, the features, regions, types, and/or classes includeone or more side branch, lumen, guidewire, stent strut, stent, jailedstents, bioresorbable vascular scaffold (BVS), drug eluting stents(DES), blooming artifact, pressure wire, guidewire, lipid,atherosclerotic plaque, stenosis, calcium, calcified plaque, calciumcontaining tissue, lesions, fat, malapposed stent; underinflated stent;over inflated stent; radio opaque marker, branching angle of arterialtree; calibration element of probe, doped films; light scatteringparticles, sheath; doped sheath; fiducial registration points, diametermeasure, calcium arc measure, thickness of region or feature ofinterest, radial measure, guide catheter, shadow region, guidewiresegment, length, and thickness and others as disclosed herein.

In one embodiment, each data element, image, and output are stored inmachine readable memory in electronic communication with the machinelearning system. In one embodiment, the set of annotated polar imagesincludes images that include one or more imaging artifacts orundesirable imaging conditions. In one embodiment, the one or moreimaging artifacts or undesirable imaging conditions are selected fromthe group includes incomplete clearing of artery prior to intravascularimaging of same; insufficient contrast; insufficient contrast solution;light intensity below an average level for intravascular imaging;contrast cloud; non-uniform rotational distortion (NURD); bloomingartifacts; jailed side branches; and reflections from imaging probecomponents. In one embodiment, method and systems disclosed herein areoperable to or perform identifying, in the predictive output images, oneor more arc-based metrics, measurements of similarity for both Ca andEEL; detected EEL diameters; and detected Ca depth. In one embodiment,the neural network is a convolution neural network, wherein number ofinput channels for first node or layer is four channels. In oneembodiment, method and systems disclosed herein are operable to orperform generating a carpet view using line projections and filteringthe carpet view to reduce noise in the predictive output images.

In one embodiment, each of the foregoing (and other examples disclosedherein relative to identifiable elements in input image data) is a datachannel that may be used as a region of interest (ROI) or feature ofinterest (FOI) to train an MLS and be detectable by the trained MLS. Inone embodiment, each of the foregoing has an associated mask or datachannel or is one element in an image mask such as ground truth mask oran output mask. In one embodiment, an output mask includes multipleregions, wherein different regions correspond to different channels,such that a multichannel segmented representation is generated relativeto the input data. In one embodiment, a first frame of image data isprocessed with the neural network of the MLS to generate a first outputmask corresponding to the first frame of image data, wherein the firstmask is modified such that regions/features of interest are identifiedwith an indicia such as color coding, hatching or otherwise. Thus, thefirst output mask includes the input image data with overlays, changesto image data or mask regions to identify its class/type, or otherindicia relative to the regions of pixels classified as being of aparticular feature, class, etc.

In one embodiment, the predictive output images include one or moreindicia indicative of boundary of predicted or classified feature. Themethod further includes converting a predictive output images from polarform to Cartesian form. Thus, an output polar image mask with indiciacorresponding to detected ROI/FOI may be converted to a Cartesian image,wherein the indicia are converted and represented in the Cartesianimage. In various embodiments, annotating and operating on Cartesianimages is avoided to generate ground truth images/masks and training ofthe neural network, and instead ground truth polar images are operatedupon and used to train a given neural network. In one embodiment, carpetview masks, and subsets thereof may be used. In one embodiment, thecarpet view has a first axis corresponding to a frame number and asecond axis corresponding to a scan line number.

In one embodiment, the neural network is a conformal neural network. Inone embodiment, the MLS includes an AI processor, wherein the AIprocessor includes one or more parallel processing elements. In oneembodiment, the AI processor includes n parallel processing elements;and further includes dedicated AI processor memory. In one embodiment,the dedicated AI processor memory ranges from about 8 GB to about 64 GB.In one embodiment, the dedicated AI processor memory ranges about 64 GBto about 128 GB. In one embodiment, the AI processor is a graphicprocessing unit. In one embodiment, the parallel processing elements areselected from the group includes of CUDA core processors, coreprocessors, tensor core processors, and stream processors. In oneembodiment, the AI processor is run locally through an edge networkappliance or server. In one embodiment, an AI processor such as agraphical processing unit is used that includes 8 GB or more ofdedicated memory in conjunction with 32 GB or more of on board RAM aspart of the computing device disposed within the housing of the datacollection/imaging systems.

In one embodiment, the method further includes reducing processing timeof MLS when classifying user image data by flattening the image databefore inputting to neural network. This may be applied during trainingphase and with regard to patient image data when classifying, detecting,and/or identifying features/regions of interest. The method furtherincludes reducing processing time of MLS when classifying user imagedata by resizing or excluding region of image before inputting to neuralnetwork. The method further includes performing a circular shift 1, 2,or 3 times with respect to one or more of the polar images. The methodfurther includes performing a left to right flip with respect to one ormore of the polar images. The method further includes performing lumendetection using an image processing method or a machine learning methodto generate a set of detected lumen boundary data. The method furtherincludes generate one or more image masks for each region or feature ofinterest identified in a patient image. Implementations of the describedtechniques may include hardware, a method or process, or computersoftware on a computer-accessible medium. The methods and systems mayinclude various busses and interface components.

One general aspect includes performing lumen detection to detect lumenboundary. In one embodiment, detected lumen boundary data, such as on aper image basis, is also input to the neural network along with patientpolar image data. Other embodiments of this aspect include correspondingcomputer systems, apparatus, and computer programs recorded on one ormore computer storage devices, each configured to perform the actions ofthe methods.

Implementations may include one or more of the following features. Inone embodiment, inputting the detected lumen boundary data reduceswaiting period for classifying regions and features of interest inpatient polar image data. Implementations of the described techniquesmay include hardware, a method or process, or computer software on acomputer-accessible medium. In one embodiment, each input image fortraining or processing patient data is transformed into multipleversions, wherein the multiple versions are generated by left rightflips and circular shifts. This provides an augmented data set, which inturns reduces error and increases accuracy of neural network/MLS. Invarious embodiments, references to a MLS also include a neural networkand vice versa.

One general aspect includes a data collection and/or imaging andregion/feature characterization system. The system also includes ahousing. The system also includes a frame grabber to receive one or moreof image data, such as polar data, ultrasound data, optical image data,x-ray image data and intravascular image data. The intravascular systemalso includes a power supply. The intravascular system also includes oneor more electronic memory storage devices in electrical communicationwith the power supply. The intravascular system also includes one ormore image processing software modules executable on the processor andstored in the one or more electronic memory storage devices. Theintravascular system also includes a computing device includes a firstprocessor, the computing device in electronic communication with thepower supply and the first processor. In one embodiment, one more AIprocessors and dedicated AI processor memory is disposed in the housingor connected thereto through one or more ports, busses, or networks. Inone embodiment, the MLS and its trained neural network is operatedremotely, such as through a client/server implementation, an edgecomputing implementation, or a cloud or software as a serviceimplementation.

In one embodiment, the system also includes one or more softwareprograms stored in the one or more electronic memory storage devices.The system also includes a machine learning system includes a neuralnetwork includes one or more machine learning software modules. Theintravascular system also includes one or more AI processors, whereinthe one or more machine learning software modules are executable on theone or more AI processors; a bus; AI processor memory; an interface tosend and receive image data from the first processor, the machinelearning system in electronic communication with the power supply,wherein the machine learning system, the computing device, and the oneor more electronic memory storage devices are disposed in the housing.In one embodiment, the bus is a PCIe bus. Other embodiments of thisaspect include corresponding computer systems, apparatus, and computerprograms recorded on one or more computer storage devices, AIprocessors, specialized ASICS, circuitry and circuitry components, eachconfigured to perform the actions of the methods. In one embodiment, thebus connects the AI processor and on board memory and processor ofdiagnostic/imaging system.

Implementations may include one or more of the following features. Thesystem wherein the housing, is the housing of an optical coherencetomography imaging system or an intravascular ultrasound imaging system.The system wherein the one or more image processing software modulesincludes one or more of: polar intravascular image to Cartesian imageconversion software, includes Cartesian intravascular image to polarimage conversion software, tissue classification overlay software tolabel regions or features of interest when displayed to an end user,lumen detection software modules, image flattening pre-processingsoftware modules, image resizing software module, image annotationsoftware with GUI for labeling or marking training images with groundtruth data, pre-processing software modules and circular shiftingsoftware modules. The system wherein the one or more machine learningsoftware modules includes one or more of: a neural network interface,lumen contour prediction, side branch prediction, image resizingmodules, user interface and input processing software modules, MLSinterface software modules to control and set parameters for neuralnetwork, MLS memory manager software, pre-processing software modules,stent strut prediction software modules, jailed stent predictionsoftware modules, guidewire prediction software modules, and interfacemodules for exchanging data with imaging system. Implementations of thedescribed techniques may include hardware, a method or process, orcomputer software on a computer-accessible medium.

In part, the disclosure relates to computer-based methods, and systemssuitable for evaluating image data from a patient on a real time orsubstantially real time basis using machine learning (ML) methods andsystems. In various embodiments, a set of image data, such a pull backof intravascular data is classified using a trained neural network suchas a convolutional neural network on a substantially real time basis. Invarious embodiments, the set of image data includes between about 400frames to about 600 frames and is obtained from memory or by imaging apatient using an imaging system. In one embodiment, a set of image datathat includes between about 400 to about 600 frames is classified. Inone embodiment, substantially real time basis ranges from about 1 secondto about 60 seconds. In one embodiment, substantially real time basisranges from about 1 second to about 30 seconds. In one embodiment,substantially real time basis ranges from about 1 second to about 20seconds. In one embodiment, substantially real time basis ranges fromabout 1 second to about 15 seconds. In one embodiment, substantiallyreal time basis ranges from about 1 second to about 10 seconds. In oneembodiment, substantially real time basis is less than about 10 seconds.In part, the disclosure is directed to improving diagnostic tools forend users such as cardiologists and imaging specialists using machinelearning techniques applied to specific problems associated withintravascular images that have Cartesian and polar representations.Further, given the use of rotating probes to obtain image data for OCT,IVUS, and other imaging data, dealing with the two coordinate systemsassociated therewith creates challenges. The present disclosureaddresses these and numerous other challenges relating to solving theproblem of quickly imaging and diagnosis a patient such that stentingand other procedures may be applied during a single session in the cathlab. The ability to perform segmentation of an image into multiplefeatures or regions of interest reduces the time a patient spends duringthe initial diagnostic procedures and subsequent treatment procedures byproviding clinician with diagnostic information to inform stentplanning, evaluation of bypass, artherectomy, and other surgicaloptions, and to assess changes in patient condition over time.

In one embodiment, MLS system training is performed using polar imagesor polar image data elements that are annotated by experts. Theannotated polar images are used to train an MLS. The MLS operates on newpolar images from a patient to generate outputs of classified imagesregions that are still in polar form. After the use of training MLS forprediction and inference, the predictive outputs from the MLS in polarform are then converted to Cartesian form and the images with classifiedtissue regions (lumen, intima, side branch, guidewire, stent, plaque,calcium, etc.) are then displayed in Cartesian form. In one embodiment,the coordinates may be revered with the images being in Cartesian formwhen annotated and then ultimately converted to polar form afterprocessing and prediction by a given MLS.

In various embodiments, probability maps and tissue maps are generatedto provide user interface feedback for various workflows. In addition,probability maps and tissue maps may be combined, compared, convolved,and otherwise used to generate output results of classifying regions andfeatures of interest using a trained neural network. In variousembodiment, a given neural network is preferably trained using annotatedpolar images.

In part, the disclosure relates to user interface designs thatfacilitate improved information and time management using one or moretissue map representation based on characterized tissue of body lumensuch as a coronary artery. In various embodiments, the various detectedROI/FOI may be co-registered with angiography data and displayed usingone or more user interfaces as part of an imaging system or otherdiagnostic system.

In part, the disclosure relates to a method for identifying regions ofinterest in a blood vessel that can include tissue types and otherfeatures such as side branches, stents, guidewires and other features,characteristics and materials of the blood vessel that uses an imagingprocessing pipeline to detect the foregoing and uses a neural network todetect other ROI/FOI such as calcium, lumen, media, intima, lipid, andothers disclosed herein.

In one embodiment, the tissue type or tissue characteristic, region ofinterest (ROI), feature of interest, classes or types or blood vesselfeature selected for segmentation and/or detection and representation inone or more mask, images, or outputs includes tissue maps includes oneor more of the following cholesterol, fiber, lipid pool, lipid,fibrofatty, calcification, calcium nodule, calcium plate, intima,thrombus, foam cells, proteoglycan, and others as disclosed herein. Thevarious systems disclosed herein are operable to perform all of themethods and processes disclosed herein using specialized circuits,controllers, FPGAs, AI processors and other components as disclosedherein.

Although, the disclosure relates to different aspects and embodiments,it is understood that the different aspects and embodiments disclosedherein can be integrated, combined, or used together as a combinationsystem, or in part, as separate components, devices, and systems, asappropriate. Thus, each embodiment disclosed herein can be incorporatedin each of the aspects to varying degrees as appropriate for a givenimplementation.

BRIEF DESCRIPTION OF DRAWINGS

The figures are not necessarily to scale, emphasis instead generallybeing placed upon illustrative principles. The figures are to beconsidered illustrative in all aspects and are not intended to limit thedisclosure, the scope of which is defined only by the claims.

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a schematic diagram of a system suitable for training a neuralnetwork using ground truth masks obtained relative to arterial imagedata such as image frames and using such as network to generatepredictive outputs classifying particular features/regions of interestaccording to an illustrative embodiment of the disclosure.

FIG. 1A is a schematic diagram of a machine learning system suitable foruse with image data, such as image data obtained relative to an artery,during a training phase according to an illustrative embodiment of thedisclosure.

FIG. 1B is a schematic diagram of a machine learning system suitable foruse with user image data, such as image data obtained relative to anartery, to detect, predict, and/or classify features of interest in userimage data according to an illustrative embodiment of the disclosure.

FIG. 1C is a schematic diagram of a machine learning system suitable foruse with user image data that shows input polar image data, such as OCTimage data, being operated upon to generate ground truth image data usedto train the MLS and predictive image data obtained using a trained MLSaccording to an illustrative embodiment of the disclosure.

FIG. 1D is a schematic diagram of a machine learning system suitable foruse with user image data that generates classified/characterizedfeatures/regions of interest relative to user image data according to anillustrative embodiment of the disclosure.

FIG. 1E is a schematic diagram of a machine learning system suitable foruse with user image data that generates classified/characterizedfeatures/regions of interest relative to user image data that includes alumen detection system and a pre-processing system according to anillustrative embodiment of the disclosure.

FIG. 1F is an exemplary process flow of a training process andsubsequent prediction process for a given illustrative embodiment of thedisclosure.

FIG. 1G shows image data from imaging a patient and three outputprobability maps corresponding to detecting media, calcium, and lumen asseparate channels given illustrative embodiment of the disclosure.

FIG. 2 is a multi-layer neural network architecture suitable fortraining using ground truth annotations and generating predictiveresults according to an illustrative embodiment of the disclosure.

FIG. 2A is a schematic diagram showing a set of outputs from a MLS thatinclude indicia corresponding to various color coded classes and the useof the foregoing to obtain projections and generate a carpet view and,optionally, a tissue map according to an illustrative embodiment of thedisclosure.

FIG. 2B is a plot of an MLS system showing loss function values for testdata and training data according to an illustrative embodiment of thedisclosure.

FIG. 2C is a plot of an MLS system showing mismatch percentage valuesfor test data and training data according to an illustrative embodimentof the disclosure.

FIG. 2D shows an intravascular polar image of a patient artery, thepolar image annotated or augmented with ground truth regions classified,and the polar image with features and/or regions of interest classifiedusing an MLS according to an illustrative embodiment of the disclosure.

FIGS. 2E-2H, from left to right, show polar image data of an artery, thepolar image annotated with ground truth lumen data, lumen detectionresults performed with an MLS having a 2D neural net, and lumendetection results performed with an MLS having a 3D neural net accordingto an illustrative embodiment of the disclosure.

FIG. 3A shows a graphical user interface of an MLS system suitable fornavigating image data and annotating the same for training the MLS or acomponent thereof according to an illustrative embodiment of thedisclosure.

FIG. 3B is a polar frame of image data from a pullback of an imagingprobe through an artery that has been annotating using graphic userinterfaced based training tools of an MLS according to an illustrativeembodiment of the disclosure.

FIG. 3C is a schematic drawing in Cartesian form of a segmented arterialimage in which tissue regions are classified by predictive operations ofMLS according to an illustrative embodiment of the disclosure.

FIGS. 3D and 3E are output images from an MLS transformed into Cartesianform that show input images and semantic segmentation of the images toshow MLS-classified media, lumen, and Ca regions/masks according toillustrative embodiments of the disclosure.

FIGS. 4A-4C are exemplary imaging and diagnostic systems that includeone or more machine learning systems that are or may be integrated withor otherwise combined with an imaging system.

FIG. 5A is a schematic diagram of a system suitable to perform guidedartherectomy using an optical imaging system, such as an OCT imagingsystem, according to illustrative embodiments of the disclosure.

FIG. 5B is a schematic diagram of a method suitable to perform guidedartherectomy using an optical imaging system, such as an OCT imagingsystem, according to illustrative embodiments of the disclosure.

FIGS. 5C and 5D are OCT images segmented to highlight the Calcium plaqueregions using a MLS system such as a deep learning based systemaccording to illustrative embodiments of the disclosure.

FIG. 5E is a user interface display of Calcium plaque on 3D, crosssection and longitudinal views according to an illustrative embodimentof the disclosure.

FIG. 5F is a calcium map created on the angiography system usingangiography co-registration according to an illustrative embodiment ofthe disclosure.

FIGS. 5G and 5H shows annotated Cartesian OCT images according to anillustrative embodiment of the disclosure.

FIG. 5I shows a magnified Cartesian OCT image that identified variouslayers of an artery according to an illustrative embodiment of thedisclosure.

FIG. 5J shows a histology image of the artery of FIG. 5C with variouslayers of the artery identified as part of a ground truth or trainingset review suitable for annotating image data for use with a given MLSembodiment according to an illustrative embodiment of the disclosure.

FIGS. 6A and 6B show a Cartesian OCT image and the predictive output ofan MLS that identifies the lumen, media, and a Ca plaques as well asshows an exemplary diameter measurement, respectively according to anillustrative embodiment of the disclosure.

FIGS. 6C to 6F show additional OCT images, ground truth masks, andoutput masks indicative of tissue classification and prediction usingthe MLS according to an according to an illustrative embodiment of thedisclosure.

FIG. 7A shows an original exemplary OCT image in Cartesian form asobtained using a combination imaging and MLS-based system according toan illustrative embodiment of the disclosure.

FIGS. 7B, 7C, and 7D shows the grayscale polar image, ground truthmasks, and deep learning output of an MLS embodiment generated with theimages of FIG. 7B as an input for annotation, training and thenprediction using the trained MLS according to an illustrative embodimentof the disclosure.

FIG. 8A shows a pre-processing operation on a polar OCT image prior tobeing annotated with expert data/ground truth data according toillustrative embodiments of the disclosure.

FIGS. 8B to 8D show various exemplary polar image masks for detectinglumen, media, and calcium according to illustrative embodiments of thedisclosure.

FIGS. 9A to 9C are polar OCT images showing circular shift and leftright flips as applied to an original OCT polar image as part of apre-processing step according to illustrative embodiments of thedisclosure.

FIGS. 10A to 10B are images showing ground truth annotated images andpredicted outputs according to illustrative embodiments of thedisclosure.

FIGS. 11A to 11D are images showing various pre-processing steps thathave been developed to increase MLS processing speed to reduce patientwaiting time according to illustrative embodiments of the disclosure.

FIGS. 12A and 12B are flow charts of exemplary processing steps by whichraw image data is routed and processed along with annotations withground truth data according to illustrative embodiments of thedisclosure.

FIGS. 13A and 13B show exemplary user interfaces for diagnostic tools tosupport stent planning through virtual stenting that incorporate tissueclassification using MLS systems and methods according to illustrativeembodiments of the disclosure.

FIG. 14 shows a representation of an artery that includes various cutplanes our boundaries from an OCT scan of a subject's artery andgeneration of a tissue map using the various sets of image data obtainedduring the pullback according to illustrative embodiments of thedisclosure.

FIG. 15 is a schematic representation of an artery obtained using one ormore imaging modalities and various cut planes suitable for generating atissue map according to an illustrative embodiment of the disclosure.

FIGS. 16-18 show various tissue map representations generated using anOCT imaging pullback of an artery with various indicia integrated into auser interface displaying the various tissue maps to support diagnosisand treatment plans such as stenting and artherectomy according toillustrative embodiments of the disclosure.

DETAILED DESCRIPTION

Various data collection and analysis systems are available to obtaininformation with regard to the coronary system. The data obtained usinga device from a blood vessel or derived data from intravascular orextravascular measurements associated therewith can be analyzed ordisplayed to assist researchers and clinicians. Optical coherencetomography (OCT) is an imaging modality that uses an interferometer toobtain distance measurements relative to a blood vessel or objectsdisposed therein. Intravascular ultrasound (IVUS) can also be used inprobes to image portions of a blood vessel. Angiography systems andfluoroscopy systems are also often used to image a patient such thatdiagnostic decisions can be made and various possible treatment optionssuch as stent placement can be carried out. These and other imagingsystems can be used to image a patient externally or internally toobtain raw data, which can include various types of image data. Thisdisclosure relates to various machine learning system (MLS) embodimentsthat include one or more networks such as neural networks to provideimproved classification of components of medical imaging data. ExemplaryMLS-based systems are shown with regard to FIGS. 1, 1A, 1B, 1C, 1D, 1E,4A, 4B, and others as disclosed and depicted herein.

In particular, image data obtained with regard to lumens of the bodysuch as coronary arteries are of particular interest. Further, given thebenefits of intravascular imaging for diagnosis, flow measurement, stentplanning and others, obtaining timely image analysis and tissue typingand classification using an MLS system is of great value. These systemsoften require extensive amounts of time to perform the image processingtasks. As such, reducing the time requirements for processing a set ofpatient image data using an MLS system is one feature of variousembodiments of the disclosure.

In general, the disclosure can apply to any intravascular datacollection devices can be used to generate and receive signals thatinclude diagnostic information, such as image data, relative to theblood vessel in which they are used. These devices can include withoutlimitation imaging devices, such as optical or ultrasound probes,pressure sensor devices, and other devices suitable for collecting datawith regard to a blood vessel or other components of a cardiovascularsystem.

FIG. 1 includes a system suitable for performing various steps of thedisclosure such as collecting image data for annotating and generatingground truth masks and for classifying using a trained neural networkand displaying such predictive results to users. Various user interfacefeatures are described herein to view and assess a visual representationof arterial information. These user interfaces can include one or moremoveable elements that can be controlled by a user with a mouse,joystick, or other control and can be operated using one or moreprocessors and memory storage elements.

During a stent delivery planning procedure, the levels and location ofapposition the user can refer to OCT and annotated angiography tofurther expand or move a stent as part of delivery planning. Thesesystem features and methods can be implemented using system 3 shown inFIG. 1.

FIG. 1 shows a system 3 which includes various data collectionsubsystems suitable for collecting data or detecting a feature of orsensing a condition of or otherwise diagnosing a subject 4. In oneembodiment, the subject is disposed upon a suitable support 19 such astable bed to chair or other suitable support. Typically, the subject 4is the human or another animal having a particular region of interest25.

The data collection system 3 includes a noninvasive imaging system suchas a nuclear magnetic resonance, x-ray, computer aided tomography, orother suitable noninvasive imaging technology. As shown as anon-limiting example of such a noninvasive imaging system, anangiography system 20 such as suitable for generating cines is shown.The angiography system 20 can include a fluoroscopy system. Angiographysystem 20 is configured to noninvasively image the subject 4 such thatframes of angiography data, typically in the form of frames of imagedata, are generated while a pullback procedure is performed using aprobe 30 such that a blood vessel in region 25 of subject 4 is imagedusing angiography in one or more imaging technologies such as OCT orIVUS, for example.

The angiography system 20 is in communication with an angiography datastorage and image management system 22, which can be implemented as aworkstation or server in one embodiment. In one embodiment, the dataprocessing relating to the collected angiography signal is performeddirectly on the detector of the angiography system 20. The images fromsystem 20 are stored and managed by the angiography data storage andimage management 22.

In one embodiment system server 50 or workstation 85 handle thefunctions of system 22. In one embodiment, the entire system 20generates electromagnetic radiation, such as x-rays. The system 20 alsoreceives such radiation after passing through the subject 4. In turn,the data processing system 22 uses the signals from the angiographysystem 20 to image one or more regions of the subject 4 including region25.

As shown in this particular example, the region of interest 25 is asubset of the vascular or peripherally vascular system such as aparticular blood vessel. This can be imaged using OCT. A catheter-baseddata collection probe 30 is introduced into the subject 4 and isdisposed in the lumen of the particular blood vessel, such as forexample, a coronary artery. The probe 30 can be a variety of types ofdata collection probes such as for example an OCT probe, an FFR probe,an IVUS probe, a probe combining features of two or more of theforegoing, and other probes suitable for imaging within a blood vessel.The probe 30 typically includes a probe tip, one or more radiopaquemarkers, an optical fiber, and a torque wire. Additionally, the probetip includes one or more data collecting subsystems such as an opticalbeam director, an acoustic beam director, a pressure detector sensor,other transducers or detectors, and combinations of the foregoing.

For a probe that includes an optical beam director, the optical fiber 33is in optical communication with the probe with the beam director. Thetorque wire defines a bore in which an optical fiber is disposed. InFIG. 1, the optical fiber 33 is shown without a torque wire surroundingit. In addition, the probe 30 also includes the sheath such as a polymersheath (not shown) which forms part of a catheter. The optical fiber 33,which in the context of an OCT system is a portion of the sample arm ofan interferometer, is optically coupled to a patient interface unit(PIU) 35 as shown.

The patient interface unit 35 includes a probe connector suitable toreceive an end of the probe 30 and be optically coupled thereto.Typically, the data collection probes 30 are disposable. The PIU 35includes suitable joints and elements based on the type of datacollection probe being used. For example a combination OCT and IVUS datacollection probe requires an OCT and IVUS PIU. The PIU 35 typically alsoincludes a motor suitable for pulling back the torque wire, sheath, andoptical fiber 33 disposed therein as part of the pullback procedure. Inaddition to being pulled back, the probe tip is also typically rotatedby the PIU 35. In this way, a blood vessel of the subject 4 can beimaged longitudinally or via cross-sections. The probe 30 can also beused to measure a particular parameter such as a fractional flow reserve(FFR) or other pressure measurement.

In turn, the PIU 35 is connected to one or more intravascular datacollection systems 40. The intravascular data collection system 40 canbe an OCT system, an IVUS system, another imaging system, andcombinations of the foregoing. For example, the system 40 in the contextof probe 30 being an OCT probe can include the sample arm of aninterferometer, the reference arm of an interferometer, photodiodes, acontrol system, and patient interface unit. Similarly, as anotherexample, in the context of an IVUS system, the intravascular datacollection system 40 can include ultrasound signal generating andprocessing circuitry, noise filters, rotatable joint, motors, andinterface units. In one embodiment, the data collection system 40 andthe angiography system 20 have a shared clock or other timing signalsconfigured to synchronize angiography video frame time stamps and OCTimage frame time stamps.

In addition to the invasive and noninvasive image data collectionsystems and devices of FIG. 1, various other types of data can becollected with regard to region 25 of the subject and other parametersof interest of the subject. For example, the data collection probe 30can include one or more pressure sensors such as for example a pressurewire. A pressure wire can be used without the additions of OCT orultrasound components. Pressure readings can be obtained along thesegments of a blood vessel in region 25 of the subject 4.

Such readings can be relayed either by a wired connection or via awireless connection. As shown in a fractional flow reserve FFR datacollection system, a wireless transceiver 47 is configured to receivepressure readings from the probe 30 and transmit them to a system togenerate FFR measurements or more locations along the measured bloodvessel. One or more displays 82, 83 can also be used to show anangiography frame of data, an OCT frame, user interfaces for OCT andangiography data and other controls and features of interest.

The intravascular image data such as the frames of intravascular datagenerated using the data collection probe 30 can be routed to the datacollection processing system 40 coupled to the probe via PIU 35. Thenoninvasive image data generated using angiography system 22 can betransmitted to, stored in, and processed by one or more servers orworkstations such as the co-registration server 50 workstation 85. Avideo frame grabber device 55 such as a computer board configured tocapture the angiography image data from system 22 can be used in variousembodiments.

In one embodiment, the server 50 includes one or more co-registrationsoftware modules 67 that are stored in memory 70 and are executed byprocessor 80. The server may includes a train neural network 52 suitablefor implementing various embodiments of the disclosures. In oneembodiment, an AI processor, such as a graphical processing unit, 53 isincluded in the server 50 and in electrical communication with memory70. The computing device/server 50 can include other typical componentsfor a processor-based computing server. Alternatively, more databasessuch as database 90 can be configured to receive image data generated,parameters of the subject, and other information generated, received byor transferred to the database 90 by one or more of the systems devicesor components shown in FIG. 1.

Although database 90 is shown connected to server 50 while being storedin memory at workstation 85, this is but one exemplary configuration.For example, the software modules 67 can be running on a processor atworkstation 85 and the database 90 can be located in the memory ofserver 50. The device or system use to run various software modules areprovided as examples. In various combinations the hardware and softwaredescribed herein can be used to obtain frames of image data, processsuch image data, and register such image data. Various software modulescan also include tissue map generation software suitable for generatingone or more tissue maps that show one or more regions of interest (ROI)and/or detected or characterized tissues or arterial material such ascalcium

As otherwise noted herein, the software modules 67 can include softwaresuch as preprocessing software, transforms, matrices, and othersoftware-based components that are used to process image data or respondto patient triggers to facilitate co-registration of different types ofimage data by other software-based components 67 or to otherwise performannotation of image data to generate ground truths and other software,modules, and functions suitable for implementing various embodiments ofthe disclosure. The modules can include lumen detection using a scanline based or image based approach, stent detection using a scan linebased or image based approach, indicator generation, apposition bargeneration for stent planning, guidewire shadow indicator to preventconfusion with dissention, side branches and missing data, and others.

The database 90 can be configured to receive and store angiography imagedata 92 such as image data generated by angiography system 20 andobtained by the frame grabber 55 server 50. The database 90 can beconfigured to receive and store OCT image data 95 such as image datagenerated by OCT system 40 and obtained by the frame grabber 55 ofserver 50.

In addition, the subject 4 can be electrically coupled via one or moreelectrodes to one more monitors such as, for example, monitor 49.Monitor 49 can include without limitation an electrocardiogram monitorconfigured to generate data relating to cardiac function and showingvarious states of the subject such as systole and diastole.

The use of arrow heads showing directionality in a given figure or thelack thereof are not intended to limit or require a direction in whichinformation can flow. For a given connector, such as the arrows andlines shown connecting the elements shown in FIG. 1, for example,information can flow in one or more directions or in only one directionas suitable for a given embodiment. The connections can include varioussuitable data transmitting connections such as optical, wire, power,wireless, or electrical connections.

One or more software modules can be used to process frames ofangiography data received from an angiography system such as system 22shown in FIG. 1. Various software modules that can include withoutlimitation software, a component thereof, or one or more steps of asoftware-based or processor executed method can be used in a givenembodiment of the disclosure.

In part, the disclosure relates to intravascular data collectionssystems and related methods by which intravascular data collected by anintravascular probe can be transformed or analyzed by a processor-basedsystem. The results of such analysis and transformation can be displayedto an end user in various representations such as a display that is incommunication with a given MLS having a neural network to classifycomponents of a medical image. In one embodiment, a given imagingsystem, such as an OCT, IVUS, x-ray based imaging system is inelectronic communication with an MLS and able to display modifiedversions of the image data obtained using a given type of imaging systemduring the same session when such image data was obtained. Variousneural network architectures may be used for image segmentation such asV-net, U-net, CUMedVision1, CUMedVision2, VGGNet, Multi-stageMulti-recursive-input Fully Convolutional Networks (M²FCN)Coarse-to-Fine Stacked Fully Convolutional Net, Deep Active LearningFramework, ResNet, combinations thereof, and other neural networks andsoftware-based machine learning frameworks suitable for imagesegmentation.

In one embodiment, the MLS includes a specialized hardware system tohandle the necessary machine learning operations and training thereofprocesses such that results can be obtained on an expedited basis suchas within from about 2 second to about 30 seconds. In one embodiment,the results are obtained in less than about 45 seconds. The specializedhardware system of a given MLS embodiment can include a plurality ofprocessors such as AI/ML processors. The machine learning system can beimplemented by training a classifier to segment or operate upon an imagesuch that its constituent tissues, tissues types, and other regions ofinterest are detected and characterized based on type or anotherparameter. In one embodiment, the lumen, intima, media and plaque aredetected and identified as having boundaries corresponding to thesedifferent tissues. Training a given MLS/neural network involves usingknown inputs and known outputs to teach the network.

The disclosure relates to an advanced machine learning system thatincludes one or more AI processors that include an increased amount ofmemory allocated on a per processor basis. The advanced machine learningsystem is designed to support a multi-channel segmentation approach.Various channels can be selected with regard to the different regions ofinterest and characteristics for a given implementation. For example, inone embodiment, a first channel, a second channel, a third channel and afourth channel are specified such that one of each of the foregoingchannels is associated with the lumen, intima, media and plaque. Otherclasses/types can be associated with different channels to facilitatesegmentation.

In one embodiment, the plaque type is classified. In some embodiments,the plaque type may be classified as calcified. In addition, given thatthe present of a plaque and other detectable features of a given sectionof an artery can indicate the presence of a constriction such as from astenosis, another feature of the disclosure is the ability to quicklyand automatically obtain one or more scores associated with a givenplaque or stenosis to help facilitate decision making by an end user.For example, a given score determined using the image data and themachine learning-based analysis thereof can help determine whether noimmediate action is recommend, or if a stent should be placed relativeto a stenosis, or if a artherectomy or other procedure such as bypass iswarranted.

For a healthy patient, arteries have various layers arranged in aconsistent structure that include the intima, media and adventitia. As aresult of the process of atherosclerosis, the intima becomespathologically thickened and may contain plaques composed of differenttypes of tissues, including fiber, proteoglycans, lipid and calcium, aswell as macrophages and other inflammatory cells. These tissue typeshave different characteristics when imaged using various imaging systemsthat can be used to establish a set of training data for one or more ofthe machine learning systems of the disclosure. The plaques that arebelieved to be most pathologically significant are the so-calledvulnerable plaques that consist of a fibrous cap with an underlyinglipid pool. Different atherosclerosis plaques have different geometricalshapes. For examples, the foam cells usually form ribbon-like featureson the shoulders of large lipid pool; the media appears like annulusaround the vessel, etc. The shape information is currently used inqualitative assessment of OCT images. In one embodiment, the neural netis trained to identify fibrous cap and/or fibrous cap with an underlyinglipid pool. In various embodiments, references to calcium herein alsoinclude calcified plaques and other calcium containing tissue, withoutlimitation.

The ability to quickly perform an imaging procedure on a patient andobtain arterial images and then processes the images using a machinelearning system while the patient is still catheterized and prepared toreceive a stent or other treatment option results in significant timesavings and improvements in patient outcomes.

The media and the outer edge of the media called External Elastic Laminaare used by physicians to size their stent during intervention. Findingthe media and measuring the diameter in a partly diseased tissue is timeconsuming and difficult. It also requires image interpretation trainingAutomatic detection and measurement of the EEL diameter addresses thesetechnical challenges faced when diagnosis or otherwise evaluating apatient for treatment options. An example of the measurement of such adiameter is shown in FIG. 6B. FIG. 3C shows a series of labeled masksand an ROI mask in a Cartesian view of an artery.

The ROI is shown as generally example and could correspond to calcium oranother feature of interest such as region containing a side branch or astent strut. Each region/feature corresponding to lumen L, intima I,plaque Q, adventitia ADV, imaging probe P, media M, and others may begenerated by the MLS using a trained NN such as a CNN. The image 385 ofFIG. 3C can be used to help guide and inform a clinician as a diagnostictool. In various embodiments, the detections of the various features inFIG. 3C are displayed using one or more user interfaces and can beco-registered with angiography data, such as shown in FIG. 5F. In someembodiments, one or more features, regions, types, classes, predictiveresults are co-registered with angiography data. In FIG. 5F, calciumplaque detected in OCT is mapped/co-registered relative to theangiography images for review and planning. In some embodiments, tissuemaps and other image data representations are used to support diagnosisand treatment recommendations. In one embodiment, each feature or classidentified, such as ADV, EEL, IEL, L, P, I, Q may be generated as a maskor a predictive mask using one or more of the trained neural networksdisclosed herein. Indicia corresponding to the output results can beshow using color coded indicia and other indicia.

In one embodiment, annotated masks regions corresponding to set or groupof pixels define a ground truth mask that are used to train one or moreneural networks disclosed herein. Once the neural network is trained,predictive or detected masks are generated that include sets of pixelsthat correspond to regions of user data as well as an identifier of thefeature or class of the region, such as whether it is lumen, calcium,EEL, or another class or feature disclosed herein. In one embodiment,predictive results are generated on a per class basis and then all ofthe predictive results for a given image data input, such as an inputframe of OCT, IVUS, or other image data, are compared on a pixel-wisebases to generate a final predictive result for all classes. In oneembodiment, the predictive results are displayed as an output image maskwith regions corresponding to a particular class so indicated by anindicia such as color and one or more legends summarizing which indiciamaps to which class.

Detecting plaque and classification of the plaque type helps thephysician in choosing their intervention strategy. They could choose toperform artherectomy if the calcium burden is too high or choose to landthe stent in a different zone depending on the underlying plaquecomposition. Automating the process of plaque identification andcharacterization eases image interpretation for the physicians andimproves their workflow

The MLS disclosed herein can be implemented with various neural networksand integrated with various imaging and diagnostic systems. In oneembodiment, the systems and methods are implemented using a deeplearning framework. In various embodiments, the MLS includes one or moreof a neural network, rules engine, fuzzy logic system, comparators,image processing modules, such as flattening, shifting, and resizingmodules, for example, heuristic systems; pattern matching and patternrecognition systems, software implementations of the foregoing.

In one embodiment, the MLS uses a neural network (NN) such as aconvolutional neural network (CNN). The CNN includes a plurality ofnodes or neurons and can receive and output image data, data derivedtherefrom, and/or changes to image data, and/or classificationinformation regarding image data components or segments. In oneembodiment, the CNN performs semantic image segmentation. In oneembodiment, semantic segmentation using a given MLS embodiment can beused to detect if image has calcium and EEL and identify the pixels withcalcium and EEL. This helps physicians solve various problems relatingto selecting treatment options and guiding a particular treatment. Inone embodiment, the MLS uses a 3D CNN such PyTorch 3D CNN or V-NET. Theneural networks described herein can be implemented using variousframeworks including PyTorch, Microsoft Cognitive Toolkit (CNTK),TensorRT, TensorFlow, and other similar proprietary and open source MLframeworks and tools.

In one embodiment, the tool selected for data annotation allows a userto select, move pixels, and/or draw boundaries. Such an exemplary userinterface tool 305 is shown in FIGS. 3A and 3B. The use of a GUIsupports consistency for annotations from frame to frame (esp. for 3D).If a standardized tool, such as a GUI-based, annotation tool is used theconstraints of the tool and set of actions for annotating supportsconsistent training data. This is in contrast to different users handannotating an image and then having the image scanned or processed. Inone embodiment, a useful feature of GUI is ability to copy data forwardfrom previous frame. Further, in one embodiment, points are annotated inpolar, saving a Cartesian to polar conversion during pre-processing.

In one embodiment, the input to the MLS includes a training set of about450 expert annotated images, where each image pixel's tissue type isannotated by one or more expert users. In one embodiment, the trainingset may be pre-processed by the same of another MLS to perform lumendetection prior to using the image data as a training set. Thus, in oneembodiment, a first set of training data is pre-preprocessed using oneor more pre-processing techniques. The pre-processing techniques caninclude lumen detection using a MLS that has been previously trainedusing a training set with annotated lumen regions or segments.The-pre-processing techniques can also be selected to speed training ofthe network and/or the predictive speed of the trained network duringbackward propagation. Accordingly, the pre-processing techniques canalso include image data flattening, a circle shift process, a circularshift process, excluding of portions of image data, such as depth databelow a noise floor, data removal can be performed on an alternatingbasis such that every other scan line of an image is removed or everyother column is removed, pixels may be filtered to remove noise andincrease uniformity of regions, and other pre-processing steps.

In one embodiment, the training set include between about 400 to about600 image data elements such as individual images or groups, clusters,or subsets of images, scan lines or pixels (“image data elements). Inone embodiment, the training set includes between about 500 to about 700images or image data elements. In one embodiment, the training setincludes between about 700 to about 800 images or image data elements.In one embodiment, the training set includes between about 800 to about900 images or image data elements. In one embodiment, the training setincludes between about 900 to about 1000 images or image data elements.In one embodiment, the training set includes between about 1000 to about1100 images or image data elements. In one embodiment, the training setinclude between about 1000 to about 5000 images or image data elements

In one embodiment, a given CNN for use with a particular MLS includes aplurality of nodes or neurons. Such a network can include variouslayers, including hidden layers. Elements of the network can have weightvalues, filter values, or bias values that change over time as thenetwork is trained to learn about input image data such as OCT, IVUS,angiography, CT scans, or other image data. In one embodiment, thenetworks used for a given MLS implementation includes a plurality offilters that change over time in response to training sets that includepolar and/or Cartesian image data inputs.

In one embodiment, a CNN interface which may include one or moregraphical user interface components is use to facilitate the batch inputof image data. A given image is an example of image data. Image data canalso include scan lines from an intravascular pullback and other sourcesof medical imaging data suitable for segmentation and classification. Inone embodiment, each image constituting image data that is a groundtruth is classified into different parts, classes or types to supportthe use of such data to train the network to detect such parts, classesor types when operating upon new patient image data. In one embodiment,the ground truth and training sets are mask generated relative toregions or features of interest that are of interest for classifying andgenerating a trained MLS to predict or detect them automatically,without human intervention, in various embodiments while an MLS isrunning Human or machine learning can be used to generate a trainingset/ground truths. Each of these can be implemented as a mask and a datachannel Each data channel is color coded in various embodiments.

As shown in FIG. 1A, a schematic representation of a system 5 forimplementing the training phase of an MLS suitable for classifyingtissue and other data types for a given set of medical image data isdepicted. Various features, regions, types, and/or classes of tissue andregions, pixels, contours and boundaries in images may be tagged oridentified relative to image data to obtain annotated image data such asground truth masks. In turn, the foregoing can be used to train a neuralnetwork such that the network can operate on image data to generateoutputs of image data with indicia suitable for classifying andvisualizing features, regions, types, and/or classes of tissue andregions, pixels, contours and boundaries using the trained network. Agiven ground truth match may include one region corresponding to aparticular channel or feature or a group of regions corresponding to aparticular channel or feature. A given region includes a set of pixelsin a given ground truth mask. In one embodiment, a ground truth mask maybe annotated to include regions or features that correspond to aparticular type or class, such as calcium, a plaque, EEL boundary, andvarious others classes, types, and classifications disclosed herein.

In one embodiment, data augmentation is performed by increasingcardinality of image data set by performing one or more transforms tothe image data, annotate image data, and/or ground truth masks. Thesetransforms may include one or more of a circular shift, left flip, rightflip, flattening, resizing, cropping, filtering, binarizing, andnormalizing. When image data is OCT data, IVUS data, or other dataobtained using one or more rotating elements, data augmentation isperformed subject to avoid transforms that inconsistent with imagingbeing performed.

In one embodiment, the MLS includes a CNN that includes one or moreinputs to receive image data and generate outputs based on the MLSoperating on the image data. In one embodiment, the MLS and/or the CNNinclude a deep learning classifier. In one embodiment, about 400 toabout 2000 image data elements, such as for example, OCT, IVUS, orangiography images, are annotated by an expert or a first MLS areprovided as image data to the MLS. This image data includes the imagedata elements and the ground truth annotations to such data elementsobtained from one or more experts and/or a first MLS for imagepre-processing, such as lumen detection. In one embodiment, each imagedata element's pixels (or a subset thereof) are annotated by an expertof a first MLS.

Accordingly, the image data that includes the image data elements andthe ground truth annotations thereto constitute a training set that areprovided as an input to the MLS. In one embodiment, the training set isinput to the MLS which includes a CNN and/or a deep learning classifier.As shown in FIG. 1A, image data/image data elements and ground truthannotations are inputs to the MLS. The output from the MLS is comparedto the ground truth annotations and/or the image data elements combinedwith the ground truth annotations and an error value is generated. Inone embodiment, the error value is the output of a cost function. In oneembodiment, the error value is a training error value. Various metrics,scores, and tests can be used to assess changes to and convergence oftraining errors over time to an acceptable threshold. In variousembodiments, Dice, Jaccard, Cosine Similarity, and Overlap distances,metrics, scores, or coefficients may be outputs, inputs, or componentsof a cost function. Additional details relating to cost functions aredescribed herein. The system of FIG. 1A and other systems disclosedherein can include various pre-processing steps to further improvetraining and or processing time.

FIG. 1F is an exemplary method 80 of the disclosure that includes atraining process and subsequent prediction process suitable for use withvarious regions/features of interest such as lumen, calcium, media, andintima. In one embodiment, the trained neural network is trained usingground truth annotations that include calcium, intima, lumen, and media.In one embodiment, the trained neural network is trained using groundtruth annotations that consist or are specifically directed to one ormore of calcium, intima, lumen, and media. The method includes acquiringimage data, which may include polar image data, such as intravasculardata. Step 100. Initially, a set of ground truth data, such as a groundtruth masks is established by reviewing and annotating the set of imagedata. In one embodiment, the method includes annotating one or moreregions or features of interest in each polar image of the set of imagessuch that each annotated region or feature is a ground truth annotation.Step 102. Pixels corresponding to a ground truth annotation may bestored in persistent electronic memory such as the database of FIG. 1.All of the data and annotations described herein may be stored indatabase or other data structure acceptable by neural network and imageprocessing software modules of a given imaging system.

In one embodiment, the ground truth annotation can be performed with auser interface as shown and discussed in more detail herein, such aswith regard to FIG. 3A. In one embodiment, the annotations are stored asground truth masks and may include a set of regions or features ofinterests that are identified by a user along with the boundary orpixels that constitute the region or features. In one embodiment, themask includes an overlay of pixels that is combined with the originalimage, such as a polar image. In one embodiment, the mask includesaddress/list of pixels that correspond to a particular ground truthannotation or predictive result.

Training a neural network of a machine learning system using set ofannotated polar images is performed. Step 104. The training may beperformed using ground truth masks and image data annotated to obtainthe ground truth masks. Ground truth masks may be augmented usingcircular flips, right flips, left flips, and other augmentationtransforms applicable to rotational imaging modalities. In part, thedisclosure relates to augmented polar images, wherein one ground truthmask is modified to increase number of ground truth masks. In oneembodiment, lumen detection is performed as an initial detection stepsuch that the ground truth masks include a lumen boundary or lumenfeature or lumen region. In one embodiment, training is performed untilone or more metrics such as a cost function or other measure of error isreduced to an acceptable level. As errors are reduced, thedetection/prediction accuracy of the MLS increases.

Once the MLS is trained, inputting image data to the neural network isperformed to generate a set of image data with predictions, detections,classifications, etc. of the various features/regions of interest. Step105. Generating K probability masks for each of K classes/types. Step106. Examples of probability map outputs/probability masks, when K is 3for three different classes is shown in FIG. 1G. Specifically, FIG. 1Gshows image data from imaging a patient and three output probabilitymaps/probability masks for different classes. The output masks/mapscorresponding to detecting media M, calcium Ca, and lumen L as separatechannels given illustrative embodiment of the disclosure. As shown, foreach respective channel, the class being detected is shown in red.Accordingly, in FIG. 1G, media M is red in media channel, the twodetected calcium regions Ca are red in the calcium channel, and lumen Lis show as red in lumen channel Other indicia can be used to color codeor represent various detected/predicted classes corresponding toregions/features of interest. In one embodiment, each class or type of afeature of interest is assigned an associated channel for tracking mask,maps, and final outputs with predictions relating to the classificationof each part of an image for which the network was trained.

If calcium, media, lumen, and intima, are the classes/types for whichthe neural network is trained to classify features/regions, K is 4. Inone embodiment, the outputs of the neural network include K probabilitymaps, such that there is one probability map for each class/type. Themethod may include generating final predictive output for each frame ofinput image data. Step 107. In one embodiment, each of the K probabilitymaps for reach of the different K classes/types, are compared on a perpixel basis and assessed such that each pixel for a given image frame isassessed and then a final predictive result is assigned to each pixel.In this way, each frame of image data is processed to generate a finalpredictive output. In one embodiment, the final predictive results arepredictive output masks that include one or more indicia correspondingto a type/class. In one embodiment, the method includes displaying finalpredictive output images from neural network/machine learning systemwith class/type indicia. Step 108.

In one embodiment, various indicia are used to color code or otherwisevisualize and show segmentation between different regions and featuresof interests identified using the trained MLS. In one embodiment,displaying output image data that has been modified to include thefeatures/regions of interest identified/predicted using the neuralnetwork/machine learning system. The foregoing steps may be performedusing one or more of the computer-based and AI processor based systemsdisclosed herein.

Various neural network architectures may be used with the embodimentsdisclosed herein. For example, V-net, U-net, CUMedVision1, CUMedVision2,VGGNet, Multi-stage Multi-recursive-input Fully Convolutional Networks(M²FCN) Coarse-to-Fine Stacked Fully Convolutional Net, Deep ActiveLearning Framework, ResNet, combinations thereof, and other neuralnetworks and software-based machine learning frameworks may be suitablefor performing feature/region of interest image segmentation andclassification as disclosed herein. FIG. 2 is an example of anarchitecture for a convolutional neural network suitable for trainingusing annotated ground truth masks and generating probability mapssuitable for assessing predictive outputs and showing classified imagedata with classified regions/features of interest. The multi-layerneural network architecture 115 of FIG. 2 is suitable for training usingground truth annotations and generating predictive results. The neuralnetwork architecture of FIG. 2 may be trained with ground truth masksfor various types/classes (calcium, lumen, media, intima, and thenumerous groups of others disclosed herein). Network 115 may be any ofthe network architectures disclosed herein or modified versions thereofor combinations thereof. In one embodiment, network 115 is aconvolutional network.

The neural network 115 include inputs 111 and outputs 117. The outputs117 are K probability maps for each image data input, when K classes arebeing specified to classify features and regions of interest. The Kprobability maps are assessed using a scoring or weighting system bywhich the output probability maps are compared for each frame of imagedata 111 and used to validate which pixels have a higher relativeprobability of being one of the K classes. As a result of theassessment, a final predictive output is generated that shows thevarious K classes and the associated features and regions with anindicia and a legend to distinguish the classes.

Various nodes are N1-N9 shown in the network 115 that have associatedsets of operations and transforms “OP” that may be applied to channelsof image data. In various implementations of network 115, the number ofchannels T is typically 16 or more channels. In an effort to streamlinethe processing speeds of network 115, Applicants have discovered thatsetting T to be 4, T=1, is suitable for classifying and training anetwork to assess arterial image data. Thus, in various embodiments, Tchannels corresponds to 4 channels, and 2T, 4T, 8T, and 16 channelscorresponds to 8 channels, 16 channels, 32 channels and 64 channels.These various channels are set at each layer/hidden layer of network tospecify how instances of the input data should be operated upon relativeto the various functions OP for the various nodes of the network. In oneembodiment, the network 115 is trained with ground truth masks usingfunction/operator such as adaptive learning algorithm. In oneembodiment, a gradient descent-based method is used to train the network115 along with the annotated ground truth masks.

The various nodes of the network 115 include sets of operations andtransforms OP for the nine nodes N1-N9 shown, OP1, OP2, OP3, OP4, OP5,OP6, OP7, OP8, and OP9. Other nodes and layers may be added betweennodes N2, N3 and nodes N7, N8 as shown the network elements 128 a, 128b. The left side of network 115 that includes nodes N1, N2, N3, and N4,and the input arrow from node N4 to N5 perform one or moredown-sampling/down converting operations DC. In contrast, on the rightside of the network 115 that includes the output arrow from node N5 toN6 and nodes N6-N9, this part of the network performs one or moreup-sampling/up converting operations UC between the nodes. In oneembodiment, the neural network architecture includes an encoder tocapture context information and of a symmetrically decoder path thatenables precise localization.

The left portion of the network 115 is the down-sampling encoder partwhere convolution blocks and down-sampling operation used to encode theinput image into feature representations at multiple different levels.The right part of the network includes up-sampling convolutionoperations and concatenation operations. The right part of the networkoperates to output an image that will have same dimension as the inputimage. In one embodiment, the network architecture has four instances ofdown-sampling operation or up-sampling operation thus the dimension ofinput image will require to be divided by 16 to avoid dimensionmismatching. In this document, image data has an M×N pixel dimension.

In one embodiment, the M×N dimension is 912×512, but other specifieddimensions may be used. In one embodiment, images other than thespecified M×N size are padded or cropped to match the specifieddimension. In various embodiments, the OP operations and functionsperfumed at each node, OP1-OP9 are selected from the group of aconvolution, a deconvolution, a additive process, a concatenationprocess, an up convert process, a down convert process, a Softmaxprocess and a PReLu process. In one embodiment, the Softmax andParametric Rectified Linear Unit (PReLu) processes are performed betweenone or more nodes as layers that transform network parameterstransmitted between nodes from input image data into probabilities forinclusion in the output probability maps. In one embodiment, the outputof the neural network include a probability assignment layer such as maybe configured using a Softmax function. The probability, on a pixel-wisebasis, or according to another grouping, such as per scan line, or permask region, based on set of trained classes, also referred to as typesor labels is provides as multiple outputs 117 for each of the K classes.In one embodiment, each node may operate as a layer or a layer may bedefined by two horizontal nodes and an arrow therebetween. In oneembodiment, the transfers between nodes shown by arrows 130 a-130 d fromleft side to the right side of the network correspond to combining orconcatenating one or more channels from left side of architecture to theright side of architecture.

In one embodiment, the network architecture of FIG. 2 is configured tocharacterize three or more types/classes of pixel labels, such ascalcium, media, lumen, and other classes and types of correspondingfeatures and regions of interests that have been images. In oneembodiment, the network 115 reduces channel count, such that T is 4, forall of the T channels shown such that processing speed of the network issubstantially real time for between about 400 and about 600 frames ofimage data. Given objective of reducing patient time in cath lab duringimaging and other procedures, a complex neural network, such as anetwork with T greater than 4 results in a slow down in the inferencetime when generating predictive results/probabilities, but also causeother problems such as overfitting in training process. In variousembodiments, setting T to be 4 in the network 115 of FIG. 2 results inimproved accuracy of network while reducing inference time. In oneembodiment, for T equal to 4, the channels of FIG. 2 are 4, 8, 16, 32,and 64 corresponding to the T channels, 2 T channels, 4T channels, 8Tchannels, and 16T channels shown.

In one embodiment, each horizontal layer, such as a given node or twonodes linked by transfer operation 130 a-130 d, is convolutional layerwhich is doing convolution operation such one or more of OP1-OP9 and/orvertical DC operations between nodes. These layers are extracting orlearning some features of the input ground truth data. For convolutionlayer at left side, various strides may be set for each node. A givenstride provides control over downsizing the image during a convolutionoperation. For example, if image size is 256×256, the output size afterconvolution will be 128×128 (256/stride). De-convolution at right sideof network performs upscaling per various UC operations.

FIG. 2A is a schematic diagram showing a set of outputs from a MLS thatinclude indicia corresponding to various color coded classes, Ca (red),lumen (blue), and media (green) and the use of the foregoing to obtainaxial projections and generate a carpet view and, optionally, a tissuemap according to an illustrative embodiment of the disclosure.

In various embodiments, the neural network architecture may be 2D or 3Dnetwork and as such, operable to process 2D data and 3D data. In oneembodiment, 3D for a pullback of F frames, wherein each frame is 2Dpolar image is displayed using a carpet view representation. In variousembodiments, filters may be applied to a 2D carpet view to remove noiseor other unwanted features such that processing time is increased whencompared with using 3D operators. Additional details relating to the useof carpet view is provided with regard to FIG. 2A.

After media and calcium detection process, each frame in polar spacewill have corresponding frames/masks for media M and calcium Caregions/features of interest. These frames/masks may include lumen L andother classes that were used to train neural network for ROI/FOIdetection. A set of four output image masks/frames 190 is shown as anexample. In various embodiments, this set of frames would includebetween about 400 to about 600 frames. Color code indicia have been usedand are shown in frames and images in FIG. 2A with blue for lumen, redfor calcium, and green for media. The output image masks may cover anentire pullback when OCT or IVUS data is used. In one embodiment, all ofthe frames of media and calcium mask, which also includes lumen in thisexample, are projected along axis, such as x-axis or y-axis as ORoperation to get a line of mask. The lines resulting from projectionoperation are shown as four lines 193. The color coding of pixels inprojected lines can be seen with green for media and red for calcium. Inone embodiment, all the lines are combined into one binary media maskand binary calcium mask based on the order of proximal to distal frame,from left to right.

This combination of line projections 193 is shown as carpet view 195.Optionally, in some embodiments, the carpet view or lines projections192 are used to create a tissue map 198 as shown. The outer most ring ofthe tissue map correspond to proximal direction, while the inner mostring shows the distal direction. In one embodiment, the tissue map 198is created by performing a polar conversion relative to the carpet view.The carpet view includes 3D data that essentially includes all framesfrom pullback. In one embodiment, a binary morphological reconstructfilter is applied to media and calcium carpet view image 198 to clean upnoise and small structures. In one embodiment, such a carpet viewfiltering step, relative to a carpet view based on predictive outputframes/masks from neural network, advantageously removes small detectedareas in carpet view image while large detected areas remain unchanged.

In one embodiment, the processed carpet view image may then be appliedto media and calcium masks such as through a convolution, additive, orcomparison process to remove noise in 3D polar space. Limitingoperations on 2D carpet view to 2D filters improves processing timerelative to operating on a 3D dataspace using 3D operators. The variousprocess and operations depicted in FIG. 2A may be implemented usingcomputer-based systems and software modules such as an image processingpipeline and neural network based feature/region detection andclassification. In one embodiment, the method of detecting ROI/FOI suchas Calcium, media, lumen, and intima using a trained neural networkincludes performing one or more of side branch detection, guidewiredetection, and stent detection using an image processing pipeline inlieu of using a trained neural network to increase processing rate ofset of image frames obtained during an OCT pullback.

In one embodiment, the carpet view is 3D representation of image framesof an intravascular pullback. The intravascular image data obtainedduring a pullback procedure using a probe can be displayed to a user bycreating a representation of the scan lines by unfolding across-sectional or polar view along a longitudinal view. A carpet viewis a two-dimensional data representation. In one embodiment, the carpetview shows a cross-sectional OCT image, but unrolled or unfolded in amanner akin to an unrolled wrapped cylinder of carpet.

The carpet view can be used to display groups of polar image data or itsunderlying components in one or more ways. For example, in oneembodiment, the carpet view collapses the radial offset dimension of the3-D intravascular data set into a single intensity. In this way, thedata can be represented in (Z, Theta) coordinates. In one embodiment,line projections can be used to generate a carpet view. In oneembodiment, the method of collapsing that radial offset dimension is tosum intensity values between a near and far offset estimate. Thisintensity value summing generates a detectable increase in contrast withrespect to certain regions in the carpet view/OCT image. These may beused to improve resolution and remove noise from output predictive masksthat are displayed using one or more panels of graphical user interfacesas shown in FIGS. 5E, and 13A, 13B.

In one embodiment, the carpet view or OCT data can be used to generate acarpet view mask. The carpet view mask of the intravascular data can begenerated to facilitate filtering of noise and artifacts from finalpredictive outputs displayed via a user interface such as shown in FIGS.5E, and 13A, 13B. The process of binarization performed relative to acarpet view image is advantageous. In one embodiment, the carpet viewmask is used as an input for subsequent intravascular image processingstages in which unwanted pixel fragments or other noise is removed.

In one embodiment, the carpet view is a two-dimensional dataset ofgenerated from scan lines of a pullback in which the dimension of theoffset along the scan line is removed or reduced. In the carpet view,the intensity values for shadows are low and the intensity values fortissues are high. The carpet view is typically a grayscale image butcolor versions can be displayed in some embodiments.

In one embodiment, trainings and experiments were performed using 3 ormore AI processors such as graphical processing units. In oneembodiment, the training code was implemented in Python using PyTorchframework. Before inputting OCT images and masks into neural network fortraining, all images were normalized to the range of (0.0˜1.0). Allimages and masks were randomized the ordering and further split into twoparts. The first part is 90% of the total dataset which was used intraining model. During training session, images and masks were randomlyshifted between (−256, +256) in vertical to augment training samples foreach epoch. The second part was used for evaluating trained model'sperformance after each epoch. In one embodiment, the cross-entropy losswas computed using a pixelwise operable Softmax function over modeloutput. The training and testing results are shown in FIGS. 2B and 2C.The loss function refers to the actual cross-entropy loss calculatedfrom prediction and ground truth as shown in FIG. 2B. Mismatchpercentage is the percentage of misclassified pixel in each imagecomparing to ground truth. Both FIGS. 2B and 2C showed the trainingprocess converged well after 300 epochs and there was no over-fittingproblem.

As noted herein, lumen detection is performed with regard to groundtruth data image data and user image data that needs features/regions ofinterest to be detected/classified. In some embodiments, lumen detectionis performed by analyzing scan lines for discontinuities, such as startand stop pairs. In other embodiments, lumen detection is implementedusing a 2D or a 3D neural network that is trained with annotated imagessuch as ground truth masks with the lumen boundary identified. FIG. 2Dshows polar image data of an artery 230 with lumen L and tissue Tidentified. FIG. 2E shows polar image data 233 annotated with groundtruth lumen data and is a mask with lumen L and tissue T correspondingto two different regions of interest/channels. FIG. 2F shows outputimage data 235 results with lumen detection performed with an MLS havinga 2D neural net. FIG. 2G shows output image data 240 results with lumendetection performed with an MLS having a 3D neural net.

FIGS. 3A and 3B show user interfaces for a system suitable fornavigating through frames of image data and annotating image data togenerate ground truths. In FIG. 3B, a polar image 350 is shown in whicha region 357 has been defined as a ground truth on a pointwise basis,such as through the movement of boundary points 360 a, 360 b, 360 c withpixels outlying the region. In one embodiment, various drawing andediting tools can be used to annotate raw image data. These annotatedimage can be used to generate ground truth masks with the class orfeature of the annotating region being defined and stored in memoryusing interface 305 of FIG. 3A. In FIG. 3A, 271 frames of anintravascular imaging pullback are available for annotation. Frame 146has been selected for annotation to provide ground truth data fortraining the MLS. The user selected region for annotation in FIG. 3Bcorresponds to Media as shown by the class identifier selected in FIG.3A. In this way, any feature/class can be selected for labelling and isstored in memory with the annotations. The annotations include pixelboundaries/regions that define masks that are matched to a particularclass or feature, such as lumen, media, calcium, etc., withoutlimitation, and may include any classes, features, or regions disclosedherein.

As an example, FIG. 2C shows a plot suitable for evaluating changes totraining error over a period of time (zero to over 600 epochs). In FIG.2C, the x-axis shows the number of training epochs and the y-axis showsthe dice coefficient, which is a measure of error. The blue and orangecurves indicate the network's performance on training and test data.

In one embodiment, the training data, such as ground truth data, alongwith the image data elements, which may include various image formats,such as raw gray scale images, is provided as input to the CNN of theMLS. The MLS is run over a period of epochs until the training error isreduced, minimized, or otherwise below a threshold. In one embodiment,the period ranges from about 100 to about 1000 epochs.

In one embodiment, the training data including the raw images and theground truth annotations are all in polar coordinates. The ground truthannotations may be augmented by existing software algorithms thatperform lumen detection. The lumen detection output from the software iscombined with expert user annotations for the media, plaque and plaquetype to generate the ground truth images. In one embodiment, the inputsto the MLS are the ground truth images and the outputs are classifiedimages. When the classified images are the same as the ground truthimages, the error is about 0.

FIG. 1B shows the use of a MLS 15 after training it using one or more ofthe training processes described herein or otherwise suitable for usewith MLS and CNNs. After a given MLS/CNN has been trained, it can thenprocess raw images and output the image segmentation that includesclassified or characterized tissue. In one embodiment, a polar imagefrom a pullback of an intravascular imaging probe is operated upon by atrained MLS to generated classified images. The system of FIG. 1B caninclude various pre-processing steps/operations 16 to further improvetraining and or processing time. In one embodiment, pre-processingincludes a normalizing step. In one embodiment, the normalizing stepincludes normalizing intensity of one or more or all of image data,ground truth annotations, masks, frames, scan lines, neural networkoutputs, and other intensity-based data such that the intensity isnormalized within a range such as from about 0 to about 1, or anotherapplicable range.

An example of an unclassified/uncharacterized OCT image in a polar formthat corresponds to an image of a cross-section of a coronary artery isshown in first image of FIG. 2D. The second image of FIG. 2D is polarimage plus a ground truth mask indicating ROIs/FOIs of media, Ca, andlumen. FIG. 2D's third image is a polar OCT image that includes aninference result or predictive result shown the color-codedindicia-green for Media, red for Calcium, and blue for lumen. An exampleof a user image data obtained using optical coherence tomography imagesthat have been classified or characterized by a trained MLS are shown inFIGS. 3D and 3E and others. In comparing FIG. 3D to FIG. 3C, it is clearthat both images are Cartesian images. In one embodiment, MLS trainingand prediction is performed on polar images and after classification hasoccurred, the polar views are converted to Cartesian views and theannotations generated by the trained MLS are displayed on the imageusing color coding or other suitable indicia or visualizations. As shownin FIG. 3D, the inner lumen region is shown as blue, the calciumcontaining plaque regions at the 9 o'clock position is shown in red andthe media region is shown roughly as a thin curved region on the rightside of the image that extends past the top and bottom midway points ofthe lumen.

A given cost function provides a metric to evaluate the output of amachine learning system by comparing the ground truth input/training setwith the expected output when operating on patient data. The goodness offit between the training data and the output data can be measured with acost function. The output of a cost function can be a value thatcorresponds to an error metric. This output is a comparative metric suchas a difference or a distance to summarize how the machine learningsystem or the neural network or other operative components thereof issucceeding in terms of accurate predictions given the predicativeresults and the ground truth used to train the system. If the outputresult of the cost function where zero, the system would be effectivelyworking perfectly. As such, iterative changes to the system can be usedto reduce the cost or error function of the system and improve itspredictive accuracy.

In addition, a pixel-wise based cost function is specified to measurethe distance or another suitable metric or score between prediction andground truth. In one embodiment, backpropagation is to update each ofthe weights in the network based values derived from the cost function.In one embodiment, on partial derivatives of the cost function are usedto update the weights during backpropagation. This weight updatingprocess has the benefit of the actual predictive results being closerthe ground truth. This has the benefit of reducing or minimizing theerror for each output neuron/node of the neural network. In oneembodiment, the neural network is a convolutional neural network (CNN).

FIG. 1C is a schematic diagram of a machine learning system 30 thatincludes a trained neural network 34, such as a network trained with anaugmented data set of annotated ground truth masks. As shown, inputimage data, such as OCT polar image data, in example shown, is operatedupon by trained neural network 34. The input polar OCT image are beingoperated upon to generated predictions/predictive outputs in the form ofpolar images that include color coded indicia corresponding to variousarterial features/regions, such as Media (green), Calcium (red), andLumen (blue) and as shown. Examples of ground truth images, in the formof annotated ground truth image masks that are used to train the neuralnetwork 34 of the MLS are also shown. In various embodiments, theoutputs of the MLS system include one or more of arc-basedmetrics/measurements of similarity for both Ca and EEL; detected EELdiameters; and detected Ca depth. In some embodiments, these values aremeasured relative to image data after classifying EEL, media, calcium,lumen, and other regions and features of interest.

Still referring to FIG. 1C, once the MLS is trained, back propagationcan be run from a time zero to a back-propagation time BPT. In this way,the predictions corresponding to annotated images or masks are generatedby inputting raw images from a patient such as from an OCT pullback intothe MLS and propagating the data through the network to generatepredictive outputs. In one embodiment, the BPT time ranges from greaterthan 0 to about 60 seconds. In one embodiment, the BPT time is less thanabout 180 seconds. In one embodiment, the BPT time is less than about 90seconds. In one embodiment, the BPT time is substantial real time asdisclosed herein, in various examples. Various neural networks 34 can beused to implement the systems and methods disclosed herein. Thesenetwork can include various hidden layers, functions, operators, andoperably combined and operator upon multiple channels. In oneembodiment, the channels correspond to the input image data, such as aset of pixels from a given frame or a subset of pixels from a givenframe.

FIG. 1D shows and MLS 55 a that works in conjunction with imageprocessing components of an imaging system to generate results that aresent to the imaging system. In one embodiment, the image processing isOCT image processing. The MLS performs AI detection using a neuralnetwork. This can be performed with an AIP/GPU. The processing time T1ranges from about 20 to about 40 seconds in one embodiment. In oneembodiment, the AIP is a GeForce GPU of the 1080 or higher family. TheAI detection can be configured to detect lumen, calcium, EEL, and any ofthe ROI/FOIs disclosed herein. The system 65 a of FIG. 1E separateslumen detection as a separate processing stages with a processing timeT3. Lumen AI detection can be performed using one or more AIPs/GPUs. Thesystem of FIG. 1E also includes various pre-processing steps to reducedownstream processing times when a patient is in the cath lap and tissuedetection/prediction is being performed while they are prepped for aprocedure. The pre-processing may have a processing time. In oneembodiment, pre-processing includes a data flattening step.

Finally, AI detection can be performed on the image data that has beenlumen detected and pre-preprocessed. The AI detection process may take atime period T4. In one embodiment, T3 ranges from about 1 to about 5seconds. In one embodiment, T4 ranges from about 5 to about 12 seconds.In one embodiment, the overall processing time T2 ranges from about 8 toabout 20 seconds. In one embodiment, T2 is from about 5 seconds to about15 seconds.

In one embodiment, the MLS and method steps and operations performed caninclude semantic image segmentation that utilize a neural network,trained end-to end, to process OCT images. Training data includes about900 OCT images in polar space with manually labeled regions of interest,wherein each region of interest corresponds to a given ground truth.Each given ground truth can map to a channel that will be detected inimage data using the trained network. In one embodiment, various inputimages and their corresponding channel specific masks are used to trainthe network using one or more algorithms. In one embodiment, astochastic gradient descent optimization algorithm is used.

In part, the disclosure relates to methods for classifying tissue anddifferent constituent layers and materials (each of the foregoingexemplary tissue types or tissue characteristics) as well as changes inthe foregoing over time. An exemplary output Cartesian image of apatient artery that has been classified by an MLS and one or morerelated methods is shown in FIGS. 3D and 3E in images B and D.

FIGS. 2D-2G show a polar image of an artery, the polar image annotatedwith ground truth lumen data, lumen detection results performed with anMLS having a 2D neural net, and lumen detection results performed withan MLS having a 3D neural net.

Various layers of a coronary artery are shown. An inner region P,corresponding to shadows or reflections from an intravascular imagingprobe P is show in the lumen of the vessel L. In particular, thedisclosure relates to detecting and identifying tissue types and regionsand/or features of interest with regard to an artery and displayingindicia or other visualization of such tissue types. This predictions orinferences from the MLS depicted in a tissue characterized/classifiedarterial image helps an end user facilitate diagnostic and treatmentdecision making. Some non-limiting examples of tissue types for whichthe methods and systems disclosed herein can be used to detect includeinner region where blood flows, the lumen, the intima, the media,external elastic lamina (EEL) (also referred to as external elasticmembrane), internal elastic lamina (IEL), adventitia, plaque, calcium orcalcified tissue, and others. The media is bounded by the IEL and EEL.The intima is bounded by the lumen and the IEL.

The disclosure relates to various embodiments that use one or moremachine learning or artificial intelligence (AI) systems to detect orsegment an image of an artery or other structure into various componenttissue types or regions of interest. In part, the machine learningsystems are designed such that they can be installed or combined with animaging system such as an intravascular imaging system, an ultrasoundsystem, or an x-ray system such as an angiography or fluoroscopy system.In one embodiment, the disclosure relates to using an MLS to performtissue characterization to detect one or more of Lumen, EEL, Media, andcalcium/calcium plaques. Additional details relating to these arteriallayers and calcium plaques are shown in FIGS. 5G-5J.

In one embodiment, the image data, which may be raw image data from oneor more imaging systems (OCT, IVUS, Angiography, X-ray, Fluoroscopy andothers), the ground truth annotations are all in one or more coordinatesystems. The ground truths, the ROI, FOIs, and the predictive resultscan each correspond to a channel and have an associated image mask in agiven training or patient image. Processing data that is in a polarcoordinate form such as OCT data is challenging for various systems.

Given the real time demands associated with generating a classified setof image data immediately after obtaining such data from a patient thatis still catheterized makes timely processing of the image data usingthe MLS a practical necessity. In order to allow machine learning and AItechniques to be used, various pre-processing or MLS design choices maybe implemented to facilitate data processing times that range from about1 second to about 20 seconds.

In general, the systems and methods disclosed herein provide to variousautomated diagnostic tools to help physicians determine if they shouldtreat a given patient and, if so, which lesion/stenosis should betreated. In one embodiment, the system provides guidance to treat themost significant lesion based on physiology. In addition, the detailsrelating to plaque type, and other MLS detected features can be used toselected shortest stent that provides maximal flow recovery. Inaddition, virtual-stenting can be implemented that provides interactiveplanning that allows for stent to be tailored for placement in an arterythat is informed by tissue classifications and other measurements asdisclosed herein.

The systems and methods disclosed herein provide guidance such that aclinician can assess various stent landing zones. This can be supportedby virtual-Stenting with virtual flow reserve values that help select alesion landing zone. Co-registration with angiography and OCT datafurther supports this. Per the use of MLS detection, normal frames(frames with healthy tissue) will have fuller Media/EEL coverage. Inthis way, MLS as implemented herein solves the problem of guiding stentplanning and helping to improve patient outcomes.

Further, automated EEL measurements obtained using the MLS describedherein inform what stent size to consider and what type of stent shouldbe used. Calcium detection via MLS provides info on lesion preparationand treatment choices, such as selecting artherectomy over stenting. Inaddition, calcium detection provides an input parameter when decidingbetween Calcium detection will help decide BVS vs DES.

FIGS. 4A-4C are exemplary imaging and diagnostic systems that includeone or more machine learning systems that are or may be integrated withor otherwise combined with an imaging system. Exemplary imaging systemscan include OCT, IVUS, angiography, fluoroscopy x-ray based imagingsystems, and others.

Referring to FIG. 4A, a data collection/imaging probe 407 and imagingsystem 410 that includes a MLS system 442 suitable for use asintravascular diagnostic system/data collection system is shown. Thesystem 442 can include an AIP such as one or more GPUs in serial orparallel and AI memory AIP. The system can be used for implementing oneor more of the AI/ML based techniques, systems, and methods describedherein. The system 410 may be used as a stent planning system forsuggesting stent placement options and provide graphical user interfacesfor simulated or virtual stent planning based on flow measurements suchas VFR, IFR, FFR, and others. Some exemplary graphical user interfacecomponents for such as system are shown in FIGS. 13A and 13B. FIG. 14Bshows a view of an artery cross-section with a measured and predictiveVFR value. In addition, a mask has been generated using a trained neuralnetwork to show Calcium (Ca/red color), media (M/green color) and lumen(L/blue color). A simulated stent is also shown in a longitudinal viewof an arterial representation in user interface shown in FIG. 13B.

In various embodiments, an intravascular probe 407 may be used to imagean artery in the presence of x-ray imaging systems such as angiography.Other imaging systems, such as CT scans, MRIs, x-ray based imagingsystems, and other 2D and 3D imaging systems may be used to generateand/or store image data in one or more memory storage devices. The probe407 in various embodiments may include other imaging modalities such as,for example, OCT, intravascular ultrasound (IVUS), and others. The probe407 is in optical communication with an intravascular diagnosticsystem/data collection system 410. The OCT optical system or subsystem431 that connects to probe 407 via an optical fiber 414 includes a lightsource such as a laser, an interferometer having a sample arm and areference arm, various optical paths, a clock generator, photodiodes,and other OCT system components.

The system 410 further includes one or more diagnostic software tools ormodules 412 relating to MLS-based image detection. This software can bestored as a non-transitory instruction on one or more memory devicessuch as memory device 445 and executed by one or more computing devicessuch as computing device 440 or MLS 442. The MLS includes one or more AIprocessors AIP and dedicated memory AIP in one embodiment. Sent planningsoftware tools can include one or more vessel profiles such as targetprofiles generated by a user, a comparator or other comparison softwareroutine for comparing pre and post stent profiles or other profiles. Ingeneral, the software 412 can process a set of intravascular data andcarry out the various methods steps described herein such as thosedescribed with regard to Figures IF, 2, 2A, 2B, 4B, 4C, 12A and 12B andother depicted and disclosed herein. In one embodiment, the software 412is stored in AIP and executable by the AIP.

The software 412 is designed to operate upon intravascular data sets andother blood vessel data from an intravascular probe or other detector ordata source such as an angiography system. In one embodiment, bloodvessel data can be recorded during a pullback procedure and stored in anelectronic memory device. The training, preprocessing, and ground truthmask generation, detected/predicted tissue classification, the neuralnets and other features and software components can be run on AIP orcomputing device 440. The software includes various MLS Training,Pre-processing, and Prediction Modules as shown. These may include

Lumen contour prediction 12A, side branch prediction 12B, resizing 12C,image flattening 12D, lumen flattening; stent strut prediction 12E, userinterface and input processing 12F, pre-processing 12G, MLS interface12H, MLS memory manager 121, GUI training module for annotating imagedata 412J to generate ground truth masks, intensity normalizing modules412K and others.

In one embodiment, software modules designed to operate uponintravascular data to characterize the tissue and identify regions ofinterest such as calcium regions, taper regions, lipid pools, and othertissue features such as. The software 412 can also compare FractionalFlow Reserve (FFR), Vascular Resistance Ratio (VRR), and other measuredand calculated intravascular data collection parameters. To the extentsuch parameters change from a stented state to a non-stent state, suchparameters can be used to generate one or more metrics.

In one embodiment, an OCT system 431 can be used. The system includes anoptical receiver such as a balanced photodiode based system receiveslight returned by the probe 407. A computing device 440, such as acomputer, a processor, an ASIC or other device that is part of thesystem 410 or is included as a separate subsystem in electrical oroptical communication with the system 410 and receives electronicsignals from the probe 7. The computing device 440 in variousembodiments includes local memory, buses and other components suitablefor processing data and utilizing software 444, such as image dataprocessing configured for stent visualization and stent malappositiondetection. In one embodiment, a PCIe bus or other high-band width, lowlatency bus is used to connect various components of a given imagingsystem, MLS, or combination system that includes both.

The stent deployment planning tools 412 can be part of or exchange datawith software 444. These tools can be used to place a virtual stent inthe lumen area that the probe 407 is disposed in relative to vesselwall. FIG. 13B shows an exemplary region of a segment of a pullbackwherein one or more virtual stents can be deployed and displayed on auser interface.

As shown, in 4A a display 446 can also be part of the system 410 forshowing information 447 such as cross-sectional and longitudinal viewsof a blood vessel generated using collected intravascular data. Once theintravascular data is obtained with the probe 407 and stored in memory445, it can be processed to generate and display information 447 such asa cross-sectional, a longitudinal, and/or a three-dimensional view ofthe blood vessel along the length of the pullback region or a subsetthereof. Two or three dimensional image masks can be used to show orstore ground truth data and predictive outcomes. These views can bedepicted as part of a user interface as shown and described below and insubsequent figures. The images of the blood vessel generated using thedistances measurements obtained from the system 410 provide informationabout the blood vessel including lumen contours, vessel diameters,vessel cross-sectional areas, landing zones, and a virtual stent boundedby the landing zones when processed using the tools and software modulesdescribed herein. In one embodiment, the MLS 442 includes one or morecomputing devices and one or more software programs or modules. Therevarious devices, components, systems, and subsystems disclosed hereinare operable to perform the tasks, methods, steps, processes and otherfeatures described herein relative to each of the foregoing.

The MLS 442 may include one or more AI processors and/or GPU and/orprocessing cores and/or stream processors and specialized memory forperforming MLS training and prediction/inference when operating onpatient image data such as image data elements. Additional details to aMLS-based system are shown in FIG. 4B.

As shown in FIG. 4B, a combined imaging and MLS system 450 is shown. Thesystem includes a display 420 that can support annotating raw image datawith points and contours to store ground truth data to train the MLS.The display can connect to MLS 425 or an imaging subsystem 430. The MLS425 and imaging system 430 have a data transfer interface 465. In oneembodiment, the MLS 425 includes one or more buses/motherboards 468 toprovide communication channels and connections for various AIPs, memory,and other components. In one embodiment, the MLS 425 includes a severalAI processors (AIP). These can be arranged in a server configuration. Asshown, three AI processors AIP1, AIP2, and AIP3 are in comminationserial, parallel, or combinations thereof. The MLS and imaging systemare disposed in a housing in one embodiment. The housing includes acooling system CS to manage the high temperatures generated by the AIPS.In one embodiment, the AIP includes one or more graphics processingunits (GPUs). In one embodiment, the housing includes a temperaturesensor TS that regulates the cooling system CS to ensure continuedoperation of the AIPs in the housing. In one embodiment, one or moreAIPs are networked to the imaging system and/or MLS and transfer dataprocessed outside of the housing.

In one embodiment, the system includes a motherboard that connects tothe AI processor. The motherboard is disposed in the housing of thesystem. The housing can be a cart that includes wheels to move thehousing and its imaging and MLS system components in the cath lab oranother location. The system includes a probe interface unit 470 thatincludes a coupler for an optical imaging probe such as an OCT probe oran IVUS probe. This PIU is also referred to as a dock in one embodiment.

In various embodiments, the trained neural network executed on an AI/MLprocessor such as a graphics processor/graphics processing unit that isdisposed in housing of imaging/data collection system. The AI/MLprocessor includes N parallel processors. In one embodiment, N rangesfrom about 2000 to about 2500. In one embodiment, N ranges from about2000 to about 3000. In one embodiment, N ranges from about 3000 to about4000. In one embodiment, N ranges from about 4000 to about 5000. In oneembodiment, N ranges from about 5000 to about 6000. Multiple parallelprocessors may be grouped on individual hardware elements referred to ascompute units. In one embodiment, the compute units range from about 20to about 80 compute units for a given AI processor. Each compute unitmay have multiple parallel processors. In one embodiment, the ComputeUnified Device Architecture (CUDA) is used with CUDA cores ranges fromabout 2000 to about 10000 cores. In one embodiment, the Compute UnifiedDevice Architecture (CUDA) is used with CUDA cores that range from about2000 to about 2500 cores. Examples of suitable parallel processorsinclude, without limitation, CUDA core and Tensor core processors fromNvidia and stream processors from AMD. In one embodiment, between about200 and about 300 Tensor Cores are included in the AI processor/graphicsprocessor used.

In one embodiment, the AI processor memory (AIP) is greater than about 8GB. In one embodiment, the AI processor memory is greater than about 16GB. In one embodiment, the AI processor memory is greater than about 32GB. In one embodiment, the AI processor memory is greater than about 64GB. In one embodiment, the AI processor memory is greater than about 128GB. In one embodiment, the AI processor memory ranges from about 4 GB toabout 256 GB. In one embodiment, the AI processor memory ranges fromabout 8 GB to about 128 GB. In one embodiment, the AI processor memoryranges from about 16 GB to about 64 GB. In one embodiment, the AIprocessor memory ranges from about 8 GB to about 32 GB. In oneembodiment, the AI processor memory ranges from about 32 GB to about 64GB. In one embodiment, one or more of the electronic memory storagedevices include an NVMe™ interface to increase processing speeds toreduce data analysis time for MLS operations when patient is in cathlab. In one embodiment, 16 GM or more of on-board RAM is used onmotherboard of imaging system/data collection system. In one embodiment,32 GB or more of on-board RAM is used on motherboard of imagingsystem/data collection system.

In one embodiment, the AI processor memory ranges from about 1 GB toabout 2 GB. In one embodiment, the AI processor memory ranges from about2 GB to about 4 GB. In one embodiment, the AI processor memory rangesfrom about 4 GB to about 6 GB. In one embodiment, the AI processormemory ranges from about 6 GB to about 8 GB. In one embodiment, the AIprocessor memory ranges from about 8 GB to about 10 GB. In oneembodiment, the AI processor memory ranges from about 10 GB to about 12GB. In one embodiment, the AI processor memory ranges from about 12 GBto about 14 GB. In one embodiment, the AI processor memory ranges fromabout 14 GB to about 16 GB.

FIG. 4C shows a system 475 with software modules suitable for operatingupon image data with two alternative process flows foridentifying/segmenting features/regions of interest relative to theimage data. In one embodiment, the image data is arterial image datasuch as polar OCT image data, but other image data from various imagesources may be used. In the upper process flow alternative, artificialintelligence-based lumen detection 487 is performed such as by using anNN trained on lumen boundaries/lumen masks as ground truths. The AIlumen detection run on an AI processor such as GPU or other processorwith dedicated memory. The image data along with masks or otherrepresentations of the detected lumen boundaries for frames of imagedata are then processed with a NN using forward and back propagation tosegment/classify the image to generate class, feature of interest,region of interest, and other detections/predictions on which the NN wastrained. As a result, predictive results such as output predictive imagemasks are generated 495.

In contrast, in other embodiments, other detections/predictions areperformed in various sequences and orders, prior to using the results ofany initial detections, such as sidebranch 479, guidewire 481, and lumen482 along with arterial image data as inputs to the MLS for processingof the arterial data and prior detections using a trained MLS. In thisway, the MLS-based detections/predictions of a given set of pixel'sclass, FOI, ROI, etc., are determined relative to image data and priorset of detections. The prior detections, such as for sidebranch,guidewire, lumen, etc., may be stored as image masks. In one embodiment,the MLS is trained on ground truth image data sets that included theprior detections implemented such as sidebranch, guidewire, and lumen,in the exemplary embodiment shown.

In part, the disclosure relates to an automated method to detect calciumcontaining tissue, such as calcified tissue, or calcium plaque, or otherregion of interest and identify the relevant nodules/region of interestfor artherectomy and using a substantially real time guided method fordoing artherectomy using a laser. In various embodiments, an OCT imagingprobe is in optical communication with an imaging laser such as a sweptsource laser. As a result, there is an optical path that extends throughthe probe to a light directing element such as a unitary lens, a GRINlens, or a beam director by which light is received from the imaginglaser, transmitted to the tissue through one of the foregoing opticalelements, and then light from the tissue is received and transmittedback to an OCT imaging system where it interferes with light generatedby the imaging laser.

FIG. 5A is a schematic diagram of an exemplary system to perform OCTguided artherectomy. In turn, FIG. 5B, a schematic diagram of anexemplary method to perform OCT guided artherectomy. In FIG. 5A, asystem 500 for detecting features of interest with an OCT system 505that includes one or more MLS systems that include feature detectionsoftware such as a trained NN is shown. An imaging probe 510 includes anoptical fiber that is part of the sample arm of an interferometer. Thereference arm of the interferometer is included in or in communicationwith the OCT system 500. A patient interface unit 470 includes a rotarycoupler or other mechanism to couple the imaging probe and its opticalfiber to the OCT system 505.

One or more optical switches 520 may be in optical communication withthe PIU 520. The optical switch 520 may be controlled by controllersoftware or a controller. In turn, the OCT system includes an imaginglaser 527 and includes or is in communication with an ablation laser.The control software or controller 535 allows the optical switch toswitch between the imaging laser for OCT imaging and the ablation laserfor targeted ablation using the GRIN lens, beam director, microlens, orother optical element for directing light in the probe 510. In turn, theMLS and its detection software, such as calcium detection software 525,can be used to identify regions of interest that contain calcium orother materials for which ablation is a preferred treatment option.

In part, the disclosure relates to calcium detection using a MLS systemsuch as a deep learning based artificial intelligence system that worksin conjunction with OCT software or other imaging software. The MLShighlights the region where there is calcium plaque or other ablationtargets. The Deep Learning network/MLS was trained using 450 to 100annotated OCT images with the calcium region marked by an expert user.This trained network was then fed with raw OCT images that weresegmented at a pixel level through the deep learning network.

In one embodiment, the calcium detection using a trained MLS system,such as a deep learning based artificial intelligence that works inconjunction with OCT software is one aspect of the disclosure. The MLShighlights the region where there is calcium plaque or other abatabletissue. The deep learning network was trained using 450 annotated OCTimages with the calcium region marked by an expert user. This trainednetwork was then fed with raw OCT images that were segmented at a pixellevel through the deep learning network.

FIGS. 5C and 5D show OCT images segmented to highlight the calciumplaque regions using a MLS system. FIG. 5E is a user interface thatdisplays a region of detected calcium such as a calcified plaque on 3D,cross section and longitudinal views. In turn, FIG. 5F is a calcium mapcreated on relative to angiography data, by co-registering OCT data withangiography data, and then displaying frames that contain calciumrelative to the angiography data. This maps the calcium plaque detectedin OCT onto the angiography graphical user interface for review andplanning. The calcium regions detected in the OCT pullback is alsomapped onto the angiography data for review by the physician. Thephysician then identifies the calcium nodules regions of interest thatare candidates for artherectomy or an ablative procedure.

As outline in FIG. 5B, a method 560 for detecting and abating tissue ofinterest using an OCT imaging probe is described. Initially, a first OCTpullback is performed 562. With the initial set of OCT data, calciumdetection is performed 565 using a MLS embodiment, or another methods ofdetecting calcium. The OCT data is collected during an angiographyimaging sessions. As a result, the OCT images and the angiography imagesare co-registered. 573 A calcium map, such as that show in FIG. 5D isgenerated. 575 Next, a clinician reviews the image data and calciumdetections and identifies plaques that warrant ablation. 577 Once theregion for artherectomy has been identified, a second OCT pullback isperformed. 579 This time during acquisition calcium detection 581 isalso performed. The imaging/ablation system uses the calcium map and thetarget calcium nodule to align the high powered laser for artherectomyas part of software-based control 583. It also performs real timecalcium detection to correct for any alignment changes between the firstOCT pullback image and the current one. This is facilitated by narrowingdown the search window of frames that need to be considered. In oneembodiment, when the catheter tip arrives at the target calcium plaquenodule, the high powered ablation laser turns on 584 to provide shortbursts of energy to vaporize and de-bulk the calcium nodule. Theablation procedure is user controlled in various embodiments to performan artherectomy 587.

FIGS. 5G and 5H shows annotated Cartesian OCT images according to anillustrative embodiment of the disclosure. They show the position of animaging probe Pin the lumen, the vessel wall W, the adventitia AD, anddetails relating to a calcium plaque and calcium angle. In oneembodiment, a calcium angle measurement can be detected and output usingthe MLS FIG. 5I shows a magnified Cartesian OCT image that identifiesvarious layers of an artery according to an illustrative embodiment ofthe disclosure.

FIG. 5J shows a histology image of the artery of FIG. 5C with variouslayers of the artery identified as part of a ground truth or trainingset review suitable for annotating image data for use with a given MLSembodiment according to an illustrative embodiment of the disclosure. InFIGS. 5G and 5H a cross-sectional image of an artery is shown that wasobtained by using an intravascular probe P shown in an in-vivoenvironment with respect to a blood vessel B having a vessel wall VWthat defines a lumen L. The elliptical/circular reflections and shadowsof the blood vessel B also includes a side branch SB.

FIGS. 6A and 6B show a Cartesian OCT image and the predictive output ofan MLS that identifies the lumen (blue indicia), media (green indicia),and Ca/Ca plaques (red indicia) as well as showing an exemplary diametermeasurement of 3.31 mm obtained using an MLS based system with a neuralnetwork (NN) trained in accordance with one or more embodimentsdisclosed herein. FIGS. 6C and 6D show an original Cartesian OCT imageand the polar image generated with scan lines collected using an imagingprobe. FIG. 6E shows ground truth masks for Ca, lumen, intima, andmedia. FIG. 6F shows the predictive results that are output from a deeplearning MLS of the disclosure.

FIG. 7A shows an original exemplary OCT image in Cartesian form asobtained using a combination imaging and MLS-based system. The imagingprobe is shown in the top right quarter of the image. The dark lumenregion is in the middle. FIG. 7B shows a grayscale polar imagecorresponding to the Cartesian image of FIG. 7A. FIGS. 7C and 7D showground truth masks, and deep learning output of an MLS embodimentgenerated with the images of FIG. 7B as an input for annotation,training and then prediction using the trained MLS. Each mask ordetection corresponds to a channel. In part, the disclosure relates to amultichannel MLS designed for use for tissue classification using imagemasks corresponding to ground truths and predictive results.

In one embodiment, semantic segmentation is performed relative tomultiple channels corresponding to a plurality of arterial features suchas tissue types, calcium, side branch, lumen, guidewire, intima, media,calcium, fibrous, stents, stent struts, stenosis, and other arterialfeatures. Ground truths for various features are used to train aconvolutional neural network. In one embodiment, two neural networks areused, such that a first neural network is used for lumen detection and asecond neural network is used to detect other arterial features afterlumen detection has been performed by a second neural network. In oneembodiment, either the first, the second, or both networks areconvolutional neural networks. In some embodiments, lumen detection isperformed relative to image data prior to detecting other features ofinterest, such as calcium, media, intima, and other features disclosedherein. Lumen detection may be implemented using various systems andmethods including those disclosed in U.S. Pat. No. 9,138,147 entitled“Lumen morphology image reconstruction based on the scan line data ofOCT,” filed on Sep. 22, 2010, the details of which are incorporated byreference in their entirety.

In one implementation, a first neural network and a first imageprocessing pipeline are used. It is advantageous to reduce patient timecatheterized during an imaging session and as such, reducing the time togenerate outputs from a classification system of regions/feature ofinterest is desirable. Accordingly, in some embodiments, a neuralnetwork is trained using frames of image data annotated with M and/or Mor more types of classes of features/regions of interests. In someembodiments, M ranges from 2 to 3. In some embodiments, M ranges from 2to 4. In some embodiments, M ranges from 2 to 5. In some embodiments, Mranges from 2 to 6. In some embodiments, M ranges from 2 to 7. In someembodiments, M ranges from 2 to 8. In some embodiments, M ranges from 2to 9. In some embodiments, M ranges from 2 to 10. In some embodiments, Mranges from 3 to 4. In some embodiments, M ranges from 3 to 5. In oneembodiment, M is 3, and the types/classes used to train the neuralnetwork are calcium, lumen, and media. The ground truthannotations/ground truth masks used to train the neural network includemasks with a group of regions annotated, wherein at least three of theannotated regions on one ground truth mask include media, calcium, andlumen annotations. In various embodiments, any collections of the typesand classes of features/regions of interest may include any of the typesand classes disclosed herein.

In various embodiments, using an image processing pipeline along with atrained neural network supports expedited processing of image datathrough each of the foregoing. In addition, training a neural network topredict features/regions of interest with annotations for all possibledetectable features in image results in one or more of excessivetraining times for network, excessive network complexity, and excessiveprocessing times when using network to predict outcomes.

FIG. 8A shows a pre-processing operation on a polar OCT image 800 priorto being annotated with expert data/ground truth data. As shown, as partof a preprocessing step, the depth pixels that do not encode or captureuseful imaging data or detectable tissue or ROIs or FOIs is effectivelyexcluded by shrinking or constraining the depth dimension to generatethe reduced, image data element 805 that can be used for iterativeaugmentations or direct annotation with expert truth data. In oneembodiment, multiple data augmentation processes are used to transformone image or image data element into multiple versions thereof such asthrough circular shifts, right left flips, and other image-basedtransformation.

For example, if an original image is circle shifted, by multiple phases,such as 90 degrees, 180 degrees, and 270 degrees, those new versions andthe original image or image data element constitute four versions. Eachof these four versions can be right left flipped to yield eight versionsor augmentations. In turn, each of these can be annotated with groundtruth annotations 810 such as by using the user interface softwaredepicted in FIGS. 3A and 3B, by which users may select sets of pixels,boundaries of pixel regions, and other geometric elements to annotateimage data with ground truth information such as classification ofcalcium, lumen, sidebranch, etc. In turn, these can also be evaluated bya trained MLS to measure detection accuracy with a cost function orother metrics. Averages of the results or statistical analysis ofmultiple versions/augmentations can be used to improve detectionaccuracy.

FIGS. 8B to 8D show various exemplary image masks for detecting lumen,media, and calcium according to illustrative embodiments of thedisclosure. Various image masks can be used such as lumen, intima,media, calcium, fibrous, and masks for any ROI or FOI disclosed herein.The masks are binary masks in one embodiment. Further, in oneembodiment, each mask can be used to reflect ground truth detections. Insome embodiments, a given ground truth mask includes one or more regionsthat correspond to a ground truth class, such as calcium, lumen, intima,etc., and other regions of the mask that do not correspond to one ormore of the classes. In one embodiment, the ground truth masks includesa first set of pixels that correspond to one or more classes and asecond set of pixels that do not correspond to the one or more classes.Ground truth masks can be used to train the neural network of a givenMLS, such as the neural network shown in FIGS. 1C and 2. In oneembodiment, resizing and other steps such as flattening and othersdescribed herein are performed before generating augmentations oradditional versions of a given raw image. A given mask can include asubset of pixels corresponding to a particular channel, wherein thechannel corresponds to a feature of interest, such as calcium, intima,and others as disclosed herein. A mask can identify pixels for a givenchannel used for training a ground truth or as a result of using atrained MLS to operate on image data to yield an output. In oneembodiment, a plurality of masks are associated with a plurality ofchannels on a one to one basis, and the processing of a given imageassigns pixels to a given channel such as a given region of interest ina mask. In one embodiment, ground truth annotations include multiplechannels of information. For example, a given ground truthannotation/mask for training the neural network includes a calciumregion, a media region, and a lumen region, and an intima region.

FIGS. 9A to 9C are polar OCT images showing circular shift and leftflips, right flips as applied to an original OCT polar image as part ofpre-processing steps performed prior to training the neural network ofMLS. In FIGS. 9A to 9C various pre-processing steps are performedrelative to image data, such as raw polar OCT images. The results ofthese processing steps or transforms is to transform one raw polar imageinto seven additional alterations or versions. This data augmentationprocess can be implemented in various ways. FIG. 9B shows a circularshift of the original image of 9A. FIG. 9C shows a left right flip ofthe transformed shifted Figure of 9B. A given augmentation such as thismay create 8 alterations of an image for a single frame. This includes 1original and 3 circular shifts (90 degrees each). The left right flipfor each of the 4 copies yields 8 in total. These operations, additionaloperations, or a subset of them can be used to pre-process data fortraining and/or prediction using the MLS.

FIGS. 10A to 10B are images showing ground truth annotated images andpredicted outputs, respectively. The inner blue ground truth math ofFIG. 10A, is bounded by the lighter media layer. The yellow layer may beencoded as another channel or classification such as lipid LP. Theorange region corresponds to calcium. In some embodiments, it isdesirable to implement a penalty or constraint system to avoid “falsepositives”. In FIG. 10B, the background channel “B” is shown. Withbackground identified in predicted results, this can be used toeffectively prevent background from being incorrectly classified as oneof the other classes or types of interest. In various embodiments,channels may be established using features that correspond to imagingartifacts, background, non-uniform rotational distortion effects, andothers, which can be generated using ground truth annotations. Thus, insome embodiments, the neural network may be trained to identifyartifacts, imaging system errors, false positives, and other imagingphenomena for which it is desirable to screen out or prevent from beingdisplayed in a given user interface representation of imaging data. Themasks showing predicted results are displayed in FIG. 10B.

FIGS. 11A to 11D are images showing various pre-processing steps thathave been developed to increase MLS processing speed to reduce patientwaiting time. As shown in FIG. 11A, the OCT image with ground truthshown via the color-coded masks (green for media, red for Ca, and bluefor lumen) reveals that along 976 pixels in vertical direction there areregions, such as region 300 along the depth dimension that lack usefuldata. The images may be flattened as shown in FIG. 11B. The white lumencontour in the middle and top regions of the polar image are flattenedto an almost straight line in FIG. 11B. This flattening allows theMLS/NN to operate on a region of the image in a transformed state thatis likely to contain relevant tissue data. Similarly, in FIG. 11C, ahalf resolution resizing is performed such as by skipping A lines (scanlines) to generate the middle image. In turn, the depth pixels in theimage are skipped or excluded to further resize the image to obtain thepre-processed image on the right size of FIG. 11C. These steps improveMLS operation and can be performed in real time or substantially realtime when a patient is catheterized after an OCT or IVUS image. In thisway, these processes can save time and improve patient outcomes. FIG.11D, shows a flattened OCT image 244 and the same flattened OCT imagewith inference or predictive results in the form of image masks showingred calcium plaque regions, blue lumen regions, and green media regions.Flattening may be performed as preprocessing step in some embodiments.This polar image can be converted to Cartesian form and displayed to anend user in view that would resemble the Cartesian image of FIG. 6B.

FIGS. 12A and 12B are flow charts of exemplary processing steps by whichraw image data is routed and processed along with annotations withground truth data. In FIG. 12A, one or more of steps A1 to A8 may beperformed using one or more computing device based systems disclosedherein. As shown, in FIG. 12A, according to one embodiment, the systemsand methods may perform one or more of the following. The imaging systemand/or MLS via a computing device or AIP may perform multipointannotation of image data or image data elements. The annotation GUIfacilities placing and connecting points to define boundary that can beconverted to image masks that effectively label a given tissue type inan image. The method may include detecting/updating lumen detectionssuch as a lumen boundary or contour. The method may include transferringannotations, lumen detections, raw image data or data elements from onesystem, such as the imaging system, to interface or intermediate systemthat provides inputs to MLS/neural network. In one embodiment, resizingimage data elements is performed to remove non-useful image data toimprove processing time. In one embodiment, the method includesgenerating binary masks for tissue labeling/annotation. Further, thedata may be formatted for input as array or in another format. Finally,the input data, which can include image mask and resized raw image dataor data elements, is input to the MLS/NN for processing such as trainingor prediction.

FIG. 12B is another embodiment and includes various steps which can beperformed by the various systems disclosed herein. In FIG. 12B, one ormore of steps A1 to A8 may be performed using one or more computingdevice based systems disclosed herein. The steps may include one or moreof the following. One or more system components, such as an imagingsystems (OCT, IVUS, x-ray, Anglo, etc.) acquires raw image data set.Various pre-processing steps can be performed. Flattening of lumen andor image regions for raw images and mask images with ground truth datamay be performed. In one embodiment, every other scan line is removed.In one embodiment, lumen contour or border is flattened to emphasizemeaningful ROI/FOI data such as shown in FIGS. 11B and 11D. In otherembodiments, images may be circle shift, right left flipped, ortransformed in different ways. This can increase entries in a giventraining set or create different versions for MLS/NN to operate on suchthat results of ML can be monitored and tested.

Clipping, skipping, or remove scan lines/data/pixels from raw images andannotated mask images to adjust rows in (polar or Cartesian form) may beperformed. For example, as shown in FIG. 11A, pixels corresponding todepth in vertical axis can be removed to resize image as shown in FIGS.11B and 11C. In addition, it may be useful to add scan lines/data/pixelto standardize columns of raw images and annotated mask images forMLS/NN processing. The method may include formatting raw image data andmask images data as needed and input to MLS/neural network forpredictive analysis or additional training

FIGS. 13A and 13B show exemplary user interfaces for diagnostic tools tosupport stent planning through virtual stenting that incorporate tissueclassification. These user interfaces help depict MLS analysis that canidentify the most significant lesion based on physiology and or theshortest stent that provides maximal flow recovery. In one embodiment,as shown by the white stent regions formed from half circles (VS) can bepositioned relative to lumen representations to perform interactivevirtual stenting. Interactive virtual stenting allows the stent to betailored to the lesion. The user can get predicted VFR values and thecurrent VFR values. The MLA view in the bottom panel, that shows percentdiameter stenosis, and the minimum lumen area, also shows VFRp and VFRvalues that change based on stenosis and/or virtual stent selection. Theangiography image shows in the top right panel is co-registered with theCartesian or polar views or longitudinal views of the artery in theother panels. In FIG. 13B, the predictive VFRp and the current VFR areshown relative to a virtual stent VS and a Cartesian arterial image thathas been masked or characterized with various indicia corresponding tolumen, calcium plagues and media ROIs/FOIs.

The methods and systems disclosed herewith provide diagnostic andplanning tools for a user. For example, the methods and systems includetools such that placement of virtual stents in an artery can beperformed automatically relative to image data from a pullback.Additional details to such diagnostic graphic user interface based toolcan be seen in FIGS. 13A and 13B. Exemplary user interfaces from acombination imaging and machine learning system suitable for suchdiagnostic planning is shown in FIGS. 13A and 13B. Further, theautomatic placements of such stents include processes, user interface,and related software-based features to display such stents at optimallocations and with the size of a suitable stent identified for an enduser.

The disclosure includes various implementations of stent planningsoftware to place a stent at an optimal location or otherwise at alocation that optimizes certain parameters. In one embodiment, theparameters optimized to facilitate stent planning include the amount offlow, which can be achieved by deploying a stent of a particular length.The proximal and distal landing zone locations for the stent and thesize of the stent are provided to an end user. These are determined byoptimizing the improvement in flow that can be achieved using a set ofpossible stents and stent deployment locations.

In one embodiment training data is created using ground truth expertguidance or guidance from a MLS such as for lumen detection. Thetraining data is separated into masks. In one embodiment, the mask isone or more individual channels. Those are used to train the trainingset. When the training set is run through the neural network each one ofthose channels is basically going to contribute to different weights andfilters of the network. In this way, the NN adapts and changes inresponse to masks/training data. In turn, when patient sample data isinput into a trained network, the raw image data is tagged or labeledwith different channels and call those channels corresponds to differentfeatures used in the training set such as images of lumen, intima,intima, media, adventitia, lumen, EEL, IEL, plaque, calcium, calciumplaques, stent, calcium, guidewires, etc.

In general, the MLS systems disclosed herein related to a multi-channelsegmentation process wherein each tissue type, region of interest,arterial layer, etc. is processed as a separate data channel with itsown image masks for generating training sets and predictive outputs.

As one exemplary approach to evaluating flow restoration as a result ofstent deployment, the methods described in U.S. patent application Ser.No. 14/115,527 entitled “METHOD AND APPARATUS FOR AUTOMATEDDETERMINATION OF A LUMEN CONTOUR OF A STENTED BLOOD VESSEL,” thecontents of which are incorporated by reference herein in theirentirety, can be used. Other approaches can be used, including asotherwise as recited herein. To understand some aspects relative to flowchanges and behaviors in an artery, it is informative to consider thefeatures shown in FIGS. 13A and 13B which show a stenosis and variousfeatures relating to the selection and position of virtual stents basedon identified landing zones and stent length(s).

In various aspects, the disclosure relates to the training of one ormore of a machine learning system, a neural network, and a convolutionalneural network using polar images of coronary arteries. In variousaspects the disclosure relates to the training of one or more of amachine learning system, a neural network, and a convolutional neuralnetwork using ground truth annotations made to polar images of coronaryarteries. In various aspects the disclosure relates to the training ofone or more of a machine learning system, a neural network, and aconvolutional neural network using ground truth annotations made topolar images of coronary arteries in which various tissue types andfeatures of interest are annotated in a training set.

In various aspects the disclosure relates to the training of one or moreof a machine learning system, a neural network, and a convolutionalneural network using ground truth annotations made to Cartesian ornon-polar images of coronary arteries in which various tissue types andfeatures/regions of interest are annotated in a training set. In variousaspects the disclosure relates to using a trained a machine learningsystem, a neural network, and/or a convolutional neural network, andcombinations thereof, to classify/characterize input patient data, inpolar or non-polar or Cartesian form to identify various tissue typesand features/regions of interest using ground truth annotations made topolar images of coronary arteries in which various tissue types andfeatures of interest are annotated in a training set.

The disclosure relates to using ground truth tissue types andfeatures/regions of interest and classifying, identifying, and orcharacterizing various tissue types and features/regions of interest inpatient image data and image data elements. In various embodiments,aspects, and for one more MLS embodiments, the tissue types and featuresof interest (FOI)/regions of interest (ROI) may include one or more ofthe Cartesian, polar, non-polar images of or images of portions of:coronary arteries, a coronary artery, OCT image, IVUS images, x-rayimages, ultrasound images, angiography images, graphs or plots of arterytrees, side branches, lumen, guidewire, stents, jailed stents,Bioresorbable Vascular Scaffold (BVS), drug eluting stents (DES),fibrous, blooming artifacts, pressure wires, lipids, calcium,atherosclerotic plaques, stenosis, plaques, calcium, calcified plaques,calcium containing tissue, lesions, fat, malapposed stents;underinflated stents; over inflated stents; radio opaque markers;branching angles of arterial trees; calibration elements such as PETdoped films; sheaths; doped sheaths; fiducial registration points,diameter changes of artery; radial changes to artery; flow measurementsobtained using imaging data; FFR values for images; and branchingmodels; combinations of the foregoing and classification or types of theforegoing.

In one embodiment, the probe includes a probe tip which includes or isin optical communication with an optical fiber. The optical fiber andthe tip of the probe are disposed within one or more sheaths such ascatheter sheath. The probe tip can include various elements such as anangled beam director or a lens cap as well as transducers for otherimaging modalities. The optical fiber of the probe 407 can also includea torque wire disposed around the fiber. The probe transmits light, inthe lumen L and receives light scattered from the vessel wall VW.

In one embodiment, the optical fiber is a portion of a sample arm of aninterferometer. A data collection probe 407, such as an OCT probe, canbe used to collect depth information suitable for imaging a sample suchas a blood vessel. For example, a set of frames of image data, such aspolar or Cartesian images shown in 3D, respectively. Figures aregenerated based upon optical signals sent and received by such a probe407. A cross-sectional image of blood vessel is formed by a collectionof scan lines as the probe rotates (see Cartesian images shown in 3D,3E, 6A, 6B. The cross-sectional image, such as images 6A, 6B is aCartesian image. FIGS. 7B and 8A show examples of polar views or polarform of medical images. Cartesian images and polar images can bereferred to frames in various embodiments.

An OCT image, such as the cross-sectional images of FIGS. 3B, 3C, and 3Dare typically acquired one scan line at a time. A sequence of samplesalong a ray originating at the catheter center to the maximum imagingdepth is referred to as a scan line in one embodiment. In oneembodiment, the smallest data unit in an OCT image is called a sample. Asequence of samples along a ray originating at the probe center to themaximum imaging depth is called a scan line. An OCT image is typicallyacquired one scan line at a time. A cross-sectional image can be formedfrom a set of scan lines collected as the probe rotates. Further, toimage a segment of an artery or other vessel, the catheter is movedlongitudinally while rotating. In this way, the probe acquires a set ofcross-sectional images in a spiral pattern. The images originate fromthe various scan lines associated with a slice of the vessel or arteryof interest. The scan lines are arranged with angles between them likespokes on a wheel. Scan lines are acquired in a polar format in oneembodiment.

User Interface Features Relating to Mapping of Tissue Features

In part, the disclosure is directed to medical diagnostic systemsincluding image navigation, user interface design, time savingenhancements and other design considerations. The foregoing features arethe disclosure were developed improve upon the problem of time andinformation management during time critical medical procedures such asthose performed in the cath lab. This disclosure describes the way thatconverting results of tissue characterization into tissue map display.Tissue characterization can be performed using various techniquesincluding pathology studies, artificial intelligence techniques, machinelearning techniques, attenuation and backscattering based techniques,and image processing detection and enhancement techniques and others asmay exist, be in development, or be developed.

The tissue map embodiments disclosed herein provides a user interfacethat enhances viewing of image data that includes various characterizedtissues and the boundaries and relative arrangement thereof with regardto one or more portions of a subject such as an artery, blood vessel, orother body lumen. Physicians, researches, technicians, and other endusers can reach planning decisions and make informed decisions basedupon diagnostic information more quickly and with a more informedcontext than would otherwise be possible given a set of images withtissue characterized regions.

Tissue characterization generally includes one or more methods todetermine the type of tissue or other tissue properties for a given setof patient tissues. Thus, if an artery is imaged, and multiple imagesform the set of image data, the tissues in each image can becharacterized to determine whether they are of a particular type such asfat, muscle, at a high level and also at a more granular level such ascalcified, intima, EEL, and other types as disclosed herein or thatotherwise exist in subjects being imaged. Typically, one or moredetection processes are used to detect the type of tissue at issue in aregion of an image. Calcium plaque mask and media mask are available fora set of image frames after machine learning inference process oranother detection process, such as an image processing pipeline-basedprocess detect calcium and media in image data. The ring like structuremay be obtained by projecting each mask along A-line as discussed abovewith regard to FIG. 2A. FIGS. 16-18 show various tissue maprepresentations generated using an OCT imaging pullback of an arterywith various indicia integrated into a user interface displaying thevarious tissue maps to support diagnosis and treatment plans such asstenting and artherectomy according to illustrative embodiments of thedisclosure.

In FIGS. 14, 15, 16, 17, and 18 each ring on the plane indicate oneframe of tissue characterization result from 3D pullback OCT data. InFIG. 14, a three-dimensional representation of an artery 1405 is shownwith a guidewire GW, regions of calcium that have been detected using aMLS or another approach and three cross-sectional boundaries R1, R2, andR3 as shown these boundaries or rings are projected into a plane togenerate a tissue map TM. Another version of a tissue map is also shownin FIG. 2A and in FIGS. 16 and 17. FIG. 15 shows a schematicrepresentation of various cut planes/rings/boundaries R1 to Rn that areshown along a 3D artery representation 1425 extending in the proximaland distal direction. In one embodiment, imaging data, such as OCT,IVUS, x-ray, or other imaging modalities may be used to generate such a3D representation of an arty 1425.

Intravascular optical coherence tomography (OCT) images providehigh-resolution visualization of coronary artery morphology. In part,the disclosure relates to the automatic detection and/or classificationof intracoronary plaques (calcium, lipid, fibrosis, and thrombus). Innerand outer calcified boundaries are also detected and displayed in oneembodiment. The process of detection and classification can enhance theinterpretation of OCT images and provide targeted information todiagnosticians. In part, the disclosure relates to systems and methodsfor displaying the results of data analysis applied to an intravasculardata set to the user in a way that is clear, easy to interpret, andconducive to diagnosing a subject such as a tissue map TM and thoseexamples shown in FIGS. 14-18.

In part, this disclosure describes a graphic user interface (GUI) thatprovides user interface and graphic data representations that can bethat generate an overall tissue map from a set of imaging data for agiven artery or another body lumen (intestine, esophagus, etc.) Thetissue map is responsive to user selections such as user selections of aparticular region of the tissue map. In response to a user clicking onor otherwise selecting a tissue map region, the user interface softwaremodules of the applicable imaging system can redirect the informationdisplayed to the user such that the underlying images or frames of imagedata specific to the use selection can be displayed.

In this way, rather than scrolling through a long series of images orframes of image data, a tissue map is presented to a user. Theinteraction and selections relative to the tissue map reduces diagnosisand analysis by expeditiously directing an end user to regions ofinterest such as calcium plagues or lipids in artery. In turn, this canexpedite stent planning while a patient is catheterized on a table andwhile blood flow has been temporarily stopped to image a section of avessel. In this way, faster decisions can be made to improve patientoutcomes. Selecting stent landing zones is also enhanced by avoidingcalcified regions in which proper stent expansion is limited by thepresence of calcium or another undesirable arterial state. Favorable andunfavorable regions of an artery can be flagged as suitable orunsuitable for stent landing zones using colors, graphics, visual cues,or other indicia or user interface features such as animations or othercues.

In part, the disclosure relates to a data collection system such as anintravascular data collection system suitable for use in cath lab suchas an optical coherence tomography system. In part, the disclosurerelates to a data collection system that includes a processor suitablefor displaying intravascular image data. The image data displayedincludes data or images generated based upon depth measurements. In oneembodiment, the image data is generated using optical coherencetomography. The system can also display a user interface for display ofintravascular information such as data relating to intravascularplaques.

In part, the diagnostic systems, methods, navigational tools, and tissuemaps and related features disclosed herein provide improvements in theform of better tools to interpret and make decisions based on plaquecomposition, lesion properties, tissue types, and related tissueinformation and blood vessel information after one or more tissuecharacterization processes and image data detection and analysissoftware modules have operated. The tissue map interfaces and organizedcharacterized tissue data from the underlying image data obtained duringa scan of a given subject demonstrate various improved image views. Forexample, in the context of OCT and other intravascular imagingmodalities a given tissue map can provide a volumetric view of a fullimaging probe pullback in single image.

In this instances, the imaging probe is pulled back through the vesselas it rotates to obtaining the imaging data. An end user can use a giventissue map to quickly realize the arc extend and thickness of calciumplagues and healthy landing zone for stent implantation. Further, otherdetails relating to tissue types can be assessed. In addition, to theextent a given tissue map reveals excessive amounts of calcium or otherundesirable tissue states, this information can facilitate an end userelecting an alternative treatment option such as a bypass orartherectomy. A given tissue map can be generated using differentmethods. A given method can vary based upon the source of the image dataobtained from scanning a patient with an imaging system.

At Pre-intervention assessment stage, physician could access plaquecomposition. As show in FIG. 16, the tissue map display 1425, pixelshaving a green color G indicate the area where media has been detectedin tissue characterization, while red color indicate the presence ofcalcium plaque Ca. The colors referenced in a given tissue map userinterface may vary and may be replaced by hatching in some instances orby using other indicia.

The dotted lines in the tissue map are shown to provide a reference torings in R1, R2, and R3 in FIG. 14 along the length of the arteryrepresentation. In one embodiment, a given tissue map may show theextent of calcium plaque. The intensity value corresponds to thethickness of calcium plaque in millimeters as shown by legend thatranges from about 0.5 mm to about 1.5 mm. Any classes or types forROIs/FOIs detected using methods disclosed herein may be displayed usinga tissue map representation such as tissue map 1425. In addition, otherinformation like fibrous and lipid containing plaque could also be addedto the tissue map such as shown in exemplary tissue map in FIG. 18. Atstent deployment stage, physician could confirm landing zone easily fromthis plot. The intensity of green color indicates that the thickness ofmedia but could be changed to indicate the distance of two EEL oppositeendpoints in 180 degree. In turn, this may be used to useful informationregarding stent size.

FIG. 17 shows tissue map of FIG. 16 with cross-sectional frames R1, R2,and R3 shown with media M, lumen L, and Calcium C identified. In oneembodiment, these regions of interest are color coded, such as withlumen as blue, Calcium as red, and media as green. In turn, FIG. 18shows a tissue map 1490 that includes indicia for guidewire GW (graycolor), media M (green color), calcium Ca (red color), lipid LP (bluecolor), and sidebranch SB (border is shown). These variousrepresentations may be used to support various workflows anddiagnostics.

In part, the disclosure relates to diagnostic systems and interfaces forthe same that facilitate navigating a blood vessel representation withrespect to which one or more imaging and tissue detection methodologieshas been applied. With respect to a given blood vessel, such as acoronary artery or other body lumen, one or more tissue types or otherregions of interest can be identified using various techniques. Inparticular, calcium nodules, calcified tissue and other calciumassociated tissues can be represented such as calcified regions in bloodvessels. One or more tissue map representations can be generated andused to displaying characterized tissue and regions of interest to auser. The characterized tissues and/or regions of interest suitable fordetection and inclusion on a one or more tissue maps can include one ormore of the following Lipid regions, lumen regions, stent struts, sidebranches, guidewires, external elastic layer (EEL), internal elasticlayer (IEL), boundaries and volumes relating to the forgoing and otherarterial features and tissues types as disclosed herein.

In part, the disclosure relates to user interface designs thatfacilitate improved information and time management using one or moretissue map representation based on characterized tissue of body lumensuch as a coronary artery. In the various tissue maps shown, moving intodirection of page is moving distally to the location in which an OCTimage probe was positioned and then pulled back. The length of pullbackis between R1 and RN, wherein RN is frame count of pullback. Lowest ringcount is most proximal. In one embodiment, high ring count is mostdistal. This can be seen in ring arrangement of FIG. 15.

The disclosure is based in part on the discovery that calcium and othertissues of a blood vessel can be detected and characterized image dataobtained with regard to a blood vessel such as OCT image data, IVUSimage data, CT scan image data, MRI image data, angiography image dataand other sources of image data. In some imaging modalities, calcifiedregions appear as discrete, darkened shapes. This is the case in OCTimages with calcium showing as darker regions relative to the brightervascular tissue background of OCT images.

In part, the disclosure relates to a method for identifying regions ofinterest in a blood vessel that can include tissue types and otherfeatures such as lumen, side branches, stents, guidewires and otherfeatures, characteristics and materials of the blood vessel.

In one embodiment, a representation of a blood vessel that has undergonetissue type analysis and/or tissue type segmentation one or more atwo-dimensional cross-section of a blood vessel or a three-dimensionallongitudinal rendering of the blood vessel. In one embodiment, therepresentation of the blood vessel or the underlying tissuecharacterized image data obtained with regard to the blood vessel istransformed into a tissue map. In one embodiment, various colors,shapes, hatching, masks, boundaries, and other graphical elements oroverlays are used to identify or segment detected tissue types and/orregions of interest in the tissue map.

In part, the disclosure relates to a system for identifying regions ofinterest in a blood vessel, the system includes: a processor incommunication with a memory, the memory containing instructions thatwhen executed cause the processor to: obtain image data of the bloodvessel; apply a plurality of filters to the image data to generate acharacteristic or type such as a tissue type. In one embodiment, theimage data is a plurality of scan lines. In one embodiment, the imagedata is an x-ray-based data. In one embodiment, the image data is apolar image. In one embodiment, one or more polar images are sampled.The samples are combined to generate a tissue characterizedrepresentation of a blood vessel. The tissue characterizedrepresentation of a blood vessel is in polar form. A tissue map isobtained in one embodiment by transforming the polar tissuecharacterized representation into a Cartesian representation. In oneembodiment, the tissue map is a series of rings, circles, or ellipsesarranged in order along a proximal to distal axis.

In part, one embodiment of the disclosure relates to an intravasculardata collection system and one or more software-based graphic userinterfaces and software modules to perform one or more detection anddisplay processes as described herein. In one embodiment, intravasculardata is collected while angiography data is simultaneously collected. Inother embodiments, angiography, CT scans, x-ray-based imaging,photography, or other imaging modalities are used to obtain imaging datawhich is used to generate a tissue map.

In part, the disclosure relates systems and methods for treatmentassessment including stent planning and surgical options by visualizinga subject's blood vessels such as one or more coronary arteries. Theimage data can be obtained using an intravascular data collection probe.The probe can be pulled back through a blood vessel and data can becollected with respect thereto. Such pullbacks and the associated datacollection are used to plan stent deployment or evaluate deployedstents. The resulting intravascular data from a pullback can be used invarious ways such as to visualize various blood vessel regions,features, and stents deployed in relation thereto. The image data usedto generate the tissue map can be co-registered with correspondingangiography data. Thus, a user can select a region of a tissue map andsee the underlying image data used to generate the map (OCT, IVUS,x-ray, etc.) and also see the angiography data with highlighting orother indicia showing the region of the blood vessel that was selectedon the tissue map.

Stents can be visualized relative to side branches in variousembodiments of the disclosure. This is an important feature as it istypically the case that during stent deployment it is desirable to avoidstenting a side branch. In this way, the tissue map can show sidebranches and the frames that contain them can be flagged as unsuitablefor use a stent landing zones. The systems and methods described hereinfacilitate visualization of stent landing zones relative to differenttypes of tissues and regions of interest. The tissue map can beco-registered with angiography data various user interface andrepresentations of stent struts and side branches based upon thedetection of these features in the intravascular data collected.

In part, the disclosure relates to intravascular data collectionsystems, such as OCT, IVUS, and other imaging modalities and thegeneration and visualization of diagnostic information such as stentlanding zones, side branches, regions of interest, and characterizedtissue regions in the blood vessel as part of a tissue map. Graphicalelements suitable for indicating diagnostic information of interest suchas the foregoing serve as user selected elements in the tissue map thatfacilitate movement to the underlying images summarized in the tissuemap.

Also disclosed herein are systems and methods for visualizing stents,tissue types, tissue volumes, and tissue boundaries. One or moresoftware modules can be used to detect side branch locations, lumencontours, and stent strut positions, generate a blood vesselrepresentation, generate a tissue map, and control navigation to imagesbased on user selections relative to the tissue map. The systems andmethods disclosed herein also include automated measurement systems andrelated features that can measure angles, thickness, volume, width,frame count, relative proximity of tissue to lumen, of various tissuetypes including calcium, lipid, fiber and others.

In various embodiments, such measurement tools can be used be used tomeasure the foregoing parameters and any geometric property for a givenregion of interest for a particular tissue type. These measurements canbe used to generate various ratings or scores suitable for considerationby end users. For example, if calcium burden in a particular region of avessel appears in tissue map but overall is only a minor amount ofsurface calcium, measurements relative thereto can help guide a user andnot exclude such a region as a candidate landing zone.

It will be appreciated that for clarity, the disclosure explicatesvarious aspects of embodiments of the applicant's teachings, whileomitting certain specific details wherever convenient or appropriate todo so. For example, discussion of like or analogous features inalternative embodiments may be somewhat abbreviated. Well-known ideas orconcepts may also for brevity not be discussed in any great detail. Theskilled person will recognize that some embodiments of the applicant'steachings may not require certain of the specifically described detailsin every implementation, which are set forth herein only to provide athorough understanding of the embodiments. Similarly, it will beapparent that the described embodiments may be susceptible to alterationor variation according to common general knowledge without departingfrom the scope of the disclosure. The detailed description ofembodiments is not to be regarded as limiting the scope of theapplicant's teachings in any manner.

The terms “about” and “substantially identical” as used herein, refer tovariations in a numerical quantity that can occur, for example, throughmeasuring or handling procedures in the real world; through inadvertenterror in these procedures; through differences/faults in the manufactureof electrical elements; through electrical losses; as well as variationsthat would be recognized by one skilled in the art as being equivalentso long as such variations do not encompass known values practiced bythe prior art. Typically, the term “about” means greater or lesser thanthe value or range of values stated by 1/10 of the stated value, e.g.,±10%. For instance, applying a voltage of about +3V DC to an element canmean a voltage between +2.7V DC and +3.3V DC. Likewise, wherein valuesare said to be “substantially identical,” the values may differ by up to5%. Whether or not modified by the term “about” or “substantially”identical, quantitative values recited in the claims include equivalentsto the recited values, e.g., variations in the numerical quantity ofsuch values that can occur, but would be recognized to be equivalents bya person skilled in the art.

Non-Limiting Software Features and Embodiments for MLS and TissueCharacterization/Classifying Systems and Methods

The following description is intended to provide an overview of devicehardware and other operating components suitable for performing themethods of the disclosure described herein. This description is notintended to limit the applicable environments or the scope of thedisclosure. Similarly, the hardware and other operating components maybe suitable as part of the apparatuses described above. The disclosurecan be practiced with other system configurations, including personalcomputers, multiprocessor systems, microprocessor-based or programmableelectronic device, network PCs, minicomputers, mainframe computers, andthe like. The disclosure can also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network such as in different roomsof a catheter or cath lab.

Some portions of the detailed description are presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations can be used by those skilled in the computer andsoftware related fields. In one embodiment, an algorithm is here, andgenerally, conceived to be a self-consistent sequence of operationsleading to a desired result. The operations performed as methods stopsor otherwise described herein are those requiring physical manipulationsof physical quantities. Usually, though not necessarily, thesequantities take the form of electrical or magnetic signals capable ofbeing stored, transferred, combined, transformed, compared, andotherwise manipulated.

Unless specifically stated otherwise as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“classifying” or “characterizing” or “correlating” or “detecting”“assessing” or “convolving” or “de-convolving” or “classifying” or“segmenting” or “training” or “annotating” or “registering” or“measuring” or “calculating” or “comparing” “generating” or “sensing” or“determining” or “displaying,” or Boolean logic or other set relatedoperations or the like, refer to the action and processes of a trainedMLS, computer system, AI processor, GPU, or electronic device, thatmanipulates and transforms data represented as physical (electronic)quantities within the computer system's or electronic devices' registersand memories into other data similarly represented as physicalquantities within electronic memories or registers or other suchinformation storage, transmission or display devices.

The present disclosure, in some embodiments, also relates to apparatusfor performing the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise a generalpurpose computer selectively activated or reconfigured by a computerprogram stored in the computer. Various circuits and components thereofcan be used to perform some of the data collection and transformationand processing described herein.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present disclosure is not described with reference toany particular programming language, and various embodiments may thus beimplemented using a variety of programming languages.

Embodiments of the disclosure may be embodied in many different forms,including, but in no way limited to, computer program logic for use witha processor (e.g., a microprocessor, microcontroller, digital signalprocessor, or general purpose computer), programmable logic for use witha programmable logic device, (e.g., a Field Programmable Gate Array(FPGA) or other programmable logic device), discrete components,integrated circuitry (e.g., an Application Specific Integrated Circuit(ASIC)), or any other means including any combination thereof. In atypical embodiment of the present disclosure, some or all of theprocessing of the data collected using an OCT probe, 2D imaging, or 3Dimaging system, and the processor-based system is implemented as a setof computer program instructions that is converted into a computerexecutable form, stored as such in a computer readable medium, andexecuted by a microprocessor under the control of an operating system.Thus, query response and input data are transformed into processorunderstandable instructions suitable for generating training sets, imagemasks, and other inputs and outputs disclosed herein. Computer programlogic implementing all or part of the functionality previously describedherein may be embodied in various forms, including, but in no waylimited to, a source code form, a computer executable form, and variousintermediate forms (e.g., forms generated by an assembler, compiler,linker, or locator). Source code may include a series of computerprogram instructions implemented in any of various programming languages(e.g., an object code, an assembly language, or a high-level languagesuch as Python, Perl, Go, FORTRAN, C, C++, JAVA, or HTML) for use withvarious operating systems or operating environments. The source code maydefine and use various data structures and communication messages. Thesource code may be in a computer executable form (e.g., via aninterpreter), or the source code may be converted (e.g., via atranslator, assembler, or compiler) into a computer executable form.

Various embodiments described herein, or components or parts thereof,may be implemented in many different embodiments of software, firmware,and/or hardware, or modules thereof. The software code or specializedcontrol hardware used to implement some of the present embodiments isnot limiting of the present invention. For example, the embodimentsdescribed hereinabove may be implemented in computer software using anysuitable computer programming language such as .NET, SQL, or MySQL,using, for example, conventional or object-oriented techniques.

Programming languages for computer software and othercomputer-implemented instructions may be translated into machinelanguage by a compiler or an assembler before execution and/or may betranslated directly at run time by an interpreter. Examples of assemblylanguages include ARM, MIPS, and x86; examples of high level languagesinclude Ada, BASIC, C, C++, C #, COBOL, Fortran, LUA, Clojure, Java,Lisp, Pascal, Object Pascal; and examples of scripting languages includeBourne script, JavaScript, Python, Ruby, PHP, and Perl.

The operation and behavior of the embodiments are described withoutspecific reference to the actual software code or specialized hardwarecomponents. The absence of such specific references is feasible becauseit is clearly understood that artisans of ordinary skill would be ableto design software and control hardware to implement the embodiments ofthe present disclosure, based on the description herein with only areasonable effort and without undue experimentation.

The various machine learning systems and associated neural networks suchas deep learning neural networks, 3D neural networks, convolutionalneural networks, 2D neural networks, N layer neural networks, feedforward neural networks, feed forward network, feed backward network,radial basis function neural network, Korhonen self-organizing neuralnetwork, recurrent neural network (RNN), modular neural network, deeplearning network, artificial intelligence-based systems and frameworks,and combinations of the foregoing.

The software for the various machine learning systems described hereinand other computer functions described herein may be implemented incomputer software using any suitable computer programming language suchas .NET, C, C++, Python, C #, Matlab programming modules and tools, andusing conventional, functional, or object-oriented techniques. Forexample, the various machine learning systems may be implemented withsoftware modules stored or otherwise maintained in computer readablemedia, e.g., RAM, ROM, secondary storage, etc. One or more processingcores (e.g., CPU, GPU and/or AI accelerator cores) of the machinelearning system may then execute the software modules to implement thefunction of the respective machine learning system (e.g., network 107,encoders 111-114, learning coach 110, etc.).

The computer program may be fixed in any form (e.g., source code form,computer executable form, or an intermediate form) either permanently ortransitorily in a tangible storage medium, such as a semiconductormemory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-ProgrammableRAM), a magnetic memory device (e.g., a diskette or fixed disk), anoptical memory device (e.g., a CD-ROM), a PC card (e.g., PCMCIA card),or other memory device. The computer program may be fixed in any form ina signal that is transmittable to a computer using any of variouscommunication technologies, including, but in no way limited to, analogtechnologies, digital technologies, optical technologies, wirelesstechnologies (e.g., Bluetooth), networking technologies, andinternetworking technologies. The computer program may be distributed inany form as a removable storage medium with accompanying printed orelectronic documentation (e.g., shrink-wrapped software), preloaded witha computer system (e.g., on system ROM or fixed disk), or distributedfrom a server or electronic bulletin board over the communication system(e.g., the Internet or World Wide Web).

Hardware logic (including programmable logic for use with a programmablelogic device) implementing all or part of the functionality previouslydescribed herein may be designed using traditional manual methods, ormay be designed, captured, simulated, or documented electronically usingvarious tools, such as Computer Aided Design (CAD), a hardwaredescription language (e.g., VHDL or AHDL), or a PLD programming language(e.g., PALASM, ABEL, or CUPL).

Programmable logic may be fixed either permanently or transitorily in atangible storage medium, such as a semiconductor memory device (e.g., aRAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memorydevice (e.g., a diskette or fixed disk), an optical memory device (e.g.,a CD-ROM), or other memory device. The programmable logic may be fixedin a signal that is transmittable to a computer using any of variouscommunication technologies, including, but in no way limited to, analogtechnologies, digital technologies, optical technologies, wirelesstechnologies (e.g., Bluetooth), networking technologies, andinternetworking technologies. The programmable logic may be distributedas a removable storage medium with accompanying printed or electronicdocumentation (e.g., shrink-wrapped software), preloaded with a computersystem (e.g., on system ROM or fixed disk), or distributed from a serveror electronic bulletin board over the communication system (e.g., theInternet or World Wide Web).

Various examples of suitable processing modules are discussed below inmore detail. As used herein a module refers to software, hardware, orfirmware suitable for performing a specific data processing or datatransmission task. Typically, in a preferred embodiment a module refersto a software routine, program, or other memory resident applicationsuitable for receiving, transforming, routing and processinginstructions, or various types of data such as resistance changes,voltage changes, current changes, guidewire-based probe data,intravascular pressure data, ratios, indices and other information ofinterest.

Computers and computer systems described herein may include operativelyassociated computer-readable media such as memory for storing softwareapplications used in obtaining, processing, storing and/or communicatingdata. It can be appreciated that such memory can be internal, external,remote or local with respect to its operatively associated computer orcomputer system.

Memory may also include any means for storing software or otherinstructions including, for example and without limitation, a hard disk,an optical disk, floppy disk, DVD (digital versatile disc), CD (compactdisc), memory stick, flash memory, ROM (read only memory), RAM (randomaccess memory), DRAM (dynamic random access memory), PROM (programmableROM), EEPROM (extended erasable PROM), and/or other likecomputer-readable media.

In general, computer-readable memory media applied in association withembodiments of the disclosure described herein may include any memorymedium capable of storing instructions executed by a programmableapparatus. Where applicable, method steps described herein may beembodied or executed as instructions stored on a computer-readablememory medium or memory media. These instructions may be softwareembodied in various programming languages such as C++, C, Java, and/or avariety of other kinds of software programming languages that may beapplied to create instructions in accordance with embodiments of thedisclosure.

A storage medium may be non-transitory or include a non-transitorydevice. Accordingly, a non-transitory storage medium or non-transitorydevice may include a device that is tangible, meaning that the devicehas a concrete physical form, although the device may change itsphysical state. Thus, for example, non-transitory refers to a deviceremaining tangible despite this change in state.

The aspects, embodiments, features, and examples of the disclosure areto be considered illustrative in all respects and are not intended tolimit the disclosure, the scope of which is defined only by the claims.Other embodiments, modifications, and usages will be apparent to thoseskilled in the art without departing from the spirit and scope of theclaimed disclosure.

The use of headings and sections in the application is not meant tolimit the disclosure; each section can apply to any aspect, embodiment,or feature of the disclosure. Only those claims which use the words“means for” are intended to be interpreted under 35 USC 112, sixthparagraph. Absent a recital of “means for” in the claims, such claimsshould not be construed under 35 USC 112. Limitations from thespecification are not intended to be read into any claims, unless suchlimitations are expressly included in the claims.

When values or ranges of values are given, each value and the end pointsof a given range and the values there between may be increased ordecreased by 20%, while still staying within the teachings of thedisclosure, unless some different range is specifically mentioned.

Throughout the application, where compositions are described as having,including, or comprising specific components, or where processes aredescribed as having, including or comprising specific process steps, itis contemplated that compositions of the present teachings also consistessentially of, or consist of, the recited components, and that theprocesses of the present teachings also consist essentially of, orconsist of, the recited process steps.

In the application, where an element or component is said to be includedin and/or selected from a list of recited elements or components, itshould be understood that the element or component can be any one of therecited elements or components and can be selected from a groupconsisting of two or more of the recited elements or components.Further, it should be understood that elements and/or features of acomposition, an apparatus, or a method described herein can be combinedin a variety of ways without departing from the spirit and scope of thepresent teachings, whether explicit or implicit herein.

The use of the terms “include,” “includes,” “including,” “have,” “has,”or “having” should be generally understood as open-ended andnon-limiting unless specifically stated otherwise.

The use of the singular herein includes the plural (and vice versa)unless specifically stated otherwise. Moreover, the singular forms “a,”“an,” and “the” include plural forms unless the context clearly dictatesotherwise. In addition, where the use of the term “about” is before aquantitative value, the present teachings also include the specificquantitative value itself, unless specifically stated otherwise.

It should be understood that the order of steps or order for performingcertain actions is immaterial so long as the present teachings remainoperable. Moreover, two or more steps or actions may be conductedsimultaneously.

Where a range or list of values is provided, each intervening valuebetween the upper and lower limits of that range or list of values isindividually contemplated and is encompassed within the disclosure as ifeach value were specifically enumerated herein. In addition, smallerranges between and including the upper and lower limits of a given rangeare contemplated and encompassed within the disclosure. The listing ofexemplary values or ranges is not a disclaimer of other values or rangesbetween and including the upper and lower limits of a given range.

It is to be understood that the figures and descriptions of thedisclosure have been simplified to illustrate elements that are relevantfor a clear understanding of the disclosure, while eliminating, forpurposes of clarity, other elements. Those of ordinary skill in the artwill recognize, however, that these and other elements may be desirable.However, because such elements are well known in the art, and becausethey do not facilitate a better understanding of the disclosure, adiscussion of such elements is not provided herein. It should beappreciated that the figures are presented for illustrative purposes andnot as construction drawings. Omitted details and modifications oralternative embodiments are within the purview of persons of ordinaryskill in the art.

It can be appreciated that, in certain aspects of the disclosure, asingle component may be replaced by multiple components, and multiplecomponents may be replaced by a single component, to provide an elementor structure or to perform a given function or functions. Except wheresuch substitution would not be operative to practice certain embodimentsof the disclosure, such substitution is considered within the scope ofthe disclosure.

The examples presented herein are intended to illustrate potential andspecific implementations of the disclosure. It can be appreciated thatthe examples are intended primarily for purposes of illustration of thedisclosure for those skilled in the art. There may be variations tothese diagrams or the operations described herein without departing fromthe spirit of the disclosure. For instance, in certain cases, methodsteps or operations may be performed or executed in differing order, oroperations may be added, deleted or modified.

What is claimed is:
 1. A method of assessing a coronary artery:acquiring a set of image data comprising frames of polar images;annotating one or more regions or features of interest in each polarimage of the set of images such that each annotated region or feature isa ground truth annotation; training a neural network of a machinelearning system using set of annotated polar images, wherein each aplurality of regions in each polar region are identified by class;inputting polar image data to the trained neural network; and displayingpredictive output images, wherein predictive output images comprisecolor coded regions, wherein each color corresponds to a class.
 2. Themethod of claim 1 wherein each image comprises a plurality of image dataelements with respect to the coronary artery.
 3. The method of claim 1annotating is performed with a graphical user interface that includesuser controls to select groups of pixels or a two-dimensional boundaryto define features of interest.
 4. The method of claim 1 wherein thetraining of the neural network is repeated until an output of a costfunction is at or below a threshold, wherein the cost function comparespredictive outputs of MLS with ground truth inputs.
 5. The method ofclaim 1 further comprising classifying the one or more annotated regionsor features of interest for each polar image as a type or class, whereintraining the neural network of a machine learning system furthercomprises classifying each annotated region by type or class.
 6. Themethod of claim 5 wherein the type or class is selected from the groupconsisting of intima, media, adventitia, lumen, EEL, IEL, plaque,calcium, calcium plaques.
 7. The method of claim 5 wherein the type orclass is selected from the group consisting of side branch, lumen,guidewire, stent strut, stent, jailed stents, bioresorbable vascularscaffold (BVS), drug eluting stents (DES), fibrous, blooming artifact,pressure wire, guidewire, lipid, atherosclerotic plaque, stenosis,calcium, calcified plaque, calcium containing tissue, lesions, fat,malapposed stent; underinflated stent; over inflated stent; radio opaquemarker, branching angle of arterial tree; calibration element of probe,doped films; light scattering particles, sheath; doped sheath; fiducialregistration points, diameter measure, radial measure, guide catheter,shadow region, guidewire segment, length, and thickness.
 8. The methodof claim 1 wherein each data element, image, and output are stored inmachine readable memory in electronic communication with the MLS.
 9. Themethod of claim 1 further comprising identifying, in the predictiveoutput images, one or more arc-based metrics, measurements of similarityfor both Ca and EEL; detected EEL diameters; and detected Ca depth. 10.The method of claim 1 wherein the neural network is a convolution neuralnetwork, wherein number of input channels for first node or layer isfour channels.
 11. The method of claim 1, wherein the predictive outputimages comprise one or more indicia indicative of boundary of predictedor classified feature.
 12. The method of claim 1 further comprisinggenerating a carpet view using line projections and filtering the carpetview to reduce noise in the predictive output images.
 13. The method ofclaim 1 wherein the neural network is a conformal neural network. 14.The method of claim 1 wherein the MLS comprises an AI processor, whereinthe AI processor comprises one or more parallel processing elements. 15.The method of claim 14, wherein the AI processor comprises N parallelprocessing elements; and further comprising dedicated AI processormemory.
 16. The method of claim 14 wherein the AI processor is a graphicprocessing unit.
 17. The method of claim 14 wherein the parallelprocessing elements are selected from the group consisting of CUDA coreprocessors, Tensor core processors, and stream processors.
 18. Themethod of claim 15, wherein the dedicated AI Processor memory rangesfrom about 8 GB to about 64 GB.
 19. The method of claim 15, wherein thededicated AI Processor memory ranges about 64 GB to about 128 GB. 20.The method of claim 1 further comprising reducing processing time of MLSwhen classifying user image data by flattening the image data beforeinputting to neural network.
 21. The method of claim 1 furthercomprising reducing processing time of MLS when classifying user imagedata by resizing or excluding region of image before inputting to neuralnetwork.
 22. The method of claim 1 further comprising augmentingtraining data by performing a circular shift 1, 2, or 3 times withrespect to one or more of the polar images.
 23. The method of claim 1further comprising performing a left to right flip with respect to oneor more of the polar images.
 24. The method of claim 1 furthercomprising removing a subset of scan lines from patient polar imageprior to inputting patient polar image data to the neural network. 25.The method of claim 1 further comprising performing lumen detectionusing an image processing method or a machine learning method togenerate a set of detected lumen boundary data.
 26. The method of claim1 further comprising performing one or more of side branch detection,guidewire detection, and stent detection using an image processingpipeline in lieu of using a trained neural network to increaseprocessing rate of set of image frames obtained during an OCT pullback.27. The method of claim 26 wherein the input of the detected lumenboundary data reduces waiting period for classifying regions andfeatures of interest in patient polar image data.
 28. The method ofclaim 1 further comprising generate one or more image masks for eachregion or feature of interest identified in a patient image.
 29. Themethod of claim 1 wherein each input image for training or processingpatient data is transformed into multiple versions, wherein the multipleversions are generated by left right flips and circular shifts.
 30. Anintravascular imaging and tissue characterization system comprising: ahousing; a frame grabber to receive one or more of x-ray image data andintravascular image data; a power supply; one or more electronic memorystorage devices in electrical communication with the power supply; oneor more image processing software modules executable on the processorand stored in the one or more electronic memory storage devices; acomputing device comprising a first processor, the computing device inelectronic communication with the power supply and the first processor;one or more software programs stored in the one or more electronicmemory storage devices; a machine learning system comprising a neuralnetwork comprising one or more machine learning software modules,wherein the neural network is trained using ground truth annotations ofpolar images, wherein the ground truth annotations comprise calcium andmedia; one or more AI processors, wherein the one or more machinelearning software modules are executable on the one or more AIprocessors, wherein the one or more AI processors comprises dedicatedmemory; and an interface to send and receive image data from the firstprocessor, the machine learning system in electronic communication withthe power supply, wherein the machine learning system, the computingdevice, and the one or more electronic memory storage devices aredisposed in the housing, wherein the trained neural network is operableto classify input image data on a substantially real time basis.
 31. Thesystem of claim 30, wherein the housing, is the housing of an opticalcoherence tomography imaging system or an intravascular ultrasoundimaging system, wherein the a substantially real time basis is less thanabout 10 seconds.
 32. The system of claim 30 wherein the one or moreimage processing software modules comprises one or more of: polarintravascular image to Cartesian image conversion software; comprisesCartesian intravascular image to polar image conversion software; tissueclassification overlay software to label regions or features of interestwhen displayed to an end user; lumen detection software modules; imageflattening pre-processing software modules; image resizing softwaremodule; image annotation software with GUI for labeling or markingtraining images with ground truth data; pre-processing software modulesand circular shifting software modules.
 33. The system of claim 30wherein the one or more machine learning software modules comprises oneor more of: a neural network interface; lumen contour prediction; sidebranch prediction; image resizing modules; user interface and inputprocessing software modules; MLS interface software modules to controland set parameters for neural network; MLS memory manager software;pre-processing software modules; stent strut prediction softwaremodules; jailed stent prediction software modules; guidewire predictionsoftware modules; and interface modules for exchanging data with imagingsystem.